In this episode, Lois Houston and Nikita Abraham, along with Nick Wagner, focus on GoldenGate’s terminology and architectural evolution.   Nick defines source and target systems, which are crucial for data replication, and then moves on to explain the data extraction and replication processes.   He also talks about the new microservices architecture, which replaces the classic architecture, offering benefits like simplified management, enhanced security, and a user-friendly interface. Nick highlights how this architecture facilitates easy upgrades and provides a streamlined experience for administrators.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------------   Episode Transcript:  

00:00

Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we’ll bring you foundational training on the most popular Oracle technologies. Let’s get started!

00:25

Nikita: Welcome to the Oracle University Podcast! I’m Nikita Abraham, Team Lead of Editorial Services with Oracle University, and with me is Lois Houston: Director of Innovation Programs.

Lois: Hi there! Thanks for joining us again as we make our way through Oracle GoldenGate 23ai. Last week, we discussed all the new features introduced in 23ai and today, we’ll move on to the terminology, the different processes and what they do, and the architecture of the product at a high level.

00:56

Nikita: Back with us is Nick Wagner, Senior Director of Product Management for Oracle GoldenGate. Hi Nick! Let’s get into some of the terminology. What do we actually call stuff in GoldenGate?

Nick: Within GoldenGate, we have our source systems and our target systems. The source is where we're going to be capturing data from, the targets, where we're going to be applying data into.

And when we start talking about things like active-active or setting up GoldenGate for high availability, where your source can also be your target, it does become a little bit more complex. And so in some of those cases, we might refer to things as East and West, or America and Europe, or different versions of that.

We also have a couple of different things within the product itself. We have what we call our Extract and our Replicat. The Extract is going to be the process that pulls the data out of the database, our capture technology. Our Replicat’s going to be the one that applies the data into the target system, or you can also look at it as a push technology.

We have what we call our Distribution Path. Our Distribution Path is going to be how we're sending the data across the network. A lot of times when customers run GoldenGate, they don't have the luxury of just having a single server of GoldenGate that can pull data from one database and push data into another one. They need to set up multiple hops of that data. And so in that case, we would use what we call a Distribution Path to send that data from one system to the next.

We also have what we call a Target Initiated Path. It's kind of a subset of your Distribution Path, but it allows you to communicate from a less secure environment into a more secure environment. 

02:33

Lois: Nick, what about parameter names. I’ve seen them in uppercase…title case…does that matter? 

Nick: GoldenGate has a lot of parameters. This is something you'll see all over the place within GoldenGate itself. 

These parameters are in your Extract and Replicat parameter files during your distribution path parameter files. Parameters for GoldenGate are case insensitive. 

Within your own environments, you can set it up to have lowercase, mixed case, whatever you want, but just be aware that they are case insensitive. GoldenGate doesn't care, it's just for readability.

And then we also have something called trail files. Trail files is where GoldenGate stores all the data before we're able to apply it into that target system. Think about it as our queuing mechanism, and we're queuing everything outside the database so that we're not overloading those database environments. And that's some of the terminology for the product itself.

We also have microservices within GoldenGate.

03:31

Nikita: And at the heart of everything is the Service Manager, right? Talk to us about what it is and what it does.

Nick: The service manager is responsible for making sure that everything else is up and running.

If you are familiar with GoldenGate classic architecture, this is kind of similar to a GoldenGate manager where that process was there to make sure that processes were running the trail files, or excuse me, that certain error logs were getting written out. If a process went down, the manager would restart that process. The service manager is performing a lot of those same functions.

Now attached to the service manager, we have our configuration service. This is new in GoldenGate 23ai. This configuration service is going to allow you to set up GoldenGate for highly available environments. So you can build HA into GoldenGate itself using the configuration service.

04:22

Lois: And what does this configuration service do?

Nick: This configuration service essentially moves the checkpoint files that used to be on disk into a database so that everything can be stored inside of a database. 

Also attached to the service manager, we have the performance metric service. This is a service that is going to be gathering all the performance metrics of GoldenGate. So it's going to tell you how fast things are going, what the latencies are, how many bytes per second we're reading from, the transaction logs or writing to our trail files. How quickly a distribution path is sending data across a network. If you want to know any of your lag information, you'll get it from the performance metrics server.

We also have the receiver service and the distribution service. These two work hand in hand to establish network communication between two GoldenGate environments. So on what we call our source system, we have a distribution service that's going to send the data to our target system.

On the target system, a receiver service is going to receive that data and then rewrite the trail files. We also have the administration service that's responsible for authentication and authorization of the users, as well as making sure that people have access to the right information.

05:33

Nikita: Ok. Moving on the deployment, how is GoldenGate actually deployed, Nick?

Nick: GoldenGate is kinda nice. So the way that the product is installed is you install the GoldenGate environment and that's what we call our service manager deployment under a specific GoldenGate home. So the software binaries themselves get installed under a home, we'll say U01/OGG23AI.

Now once I've installed GoldenGate once, that's my OGG home. I can now have any number of service managers and deployments tied to that same home. 

06:11

Lois: Ok, let’s work with an example to make this simpler. Let’s say I've got a service manager that's going be responsible for three different deployments: Accounting, Finance, and Sales.

Nick: Each of these deployments is going to reside in its own directory. Each of these deployments is going to have its own set of microservices.

And so this also means that each of these deployments can have their own set of users. So the people that access the GoldenGate accounting deployment can be different than the ones that access the sales deployment. This means with this distribution of roles that I can have somebody come in and administer the sales database, but they wouldn't have any information or any access to accounting or finance.

And this is very important, it allows you to really pull that information apart and separate it. Each of these environments also has their own set of parameter files, Extract process, Replicat process, distribution services, and everything. So it's a very nice way of splitting things up, but all having them tied to the same GoldenGate home system.

And this home is very important. So I can take a deployment, let's say my finance deployment, and if I want to move it to a new GoldenGate home and that GoldenGate home is a different version, like let's say that my original home is 23.4, my new GoldenGate home is 23.7, I simply stop that GoldenGate deployment.

I stopped at a finance deployment. I changed its OGG home from 23.4 to 23.7. I restart the deployment, that deployment is automatically upgraded to the new environment and attached to the new system. So it makes upgrading very, very simple, very easy, very elegant.

07:53

Nikita: Ok. So, we’ve spoken about the services…some of the terminology. Let’s get into the architecture next.

Nick: So when we talk about the architecture for GoldenGate, we used to have two different architectures. We had a classic architecture and a microservices architecture. Classic architecture was something that's been around since the very beginning of GoldenGate in the late '90s.

We announced that, that architecture was deprecated in 19c. And Oracle deprecated means that feature is no longer going to be enhanced and it'll be patched selectively. And at some point in the future, it'll be entirely desupported.

Well, GoldenGate 23ai is that future. And so in 23ai, the classic architecture is desupported, that means that it's no longer in the build at all. And so it's just microservices architecture.

08:41

Lois: Is there a tool to assist with this migration? 

Nick: We do have a migration utility that will convert an old classic architecture into the new microservices architecture.

But there is quite a bit of learning curve to the new microservices architecture. So it's important that we go through how it works in the changes.

09:04

Are you looking to optimize your implementation strategies and improve efficiency? We have a solution for you! Our new Oracle Fusion Cloud Applications Foundations training and certification program. You’ll learn to leverage Oracle Modern Best Practice (OMBP) to re-imagine business processes using advanced technologies in Oracle Fusion Cloud Applications such as AI, mobile, analytics, and more. Visit mylearn.oracle.com to get started today. 

09:37

Nikita: Welcome back! Nick, what are the benefits of this microservices architecture? 

Nick: It's got that simplified lifecycle for patching and upgrading.

A lot of the GoldenGate patches that you get, especially these bundle patches, are complete installs as well. So you can go into My Oracle Support and download a complete install of a patch and that way, you don't have to use old patch to apply them. The only time you'll be using old patch is for one-off patches or smaller patches that need to be applied to your GoldenGate system.

The microservices product has the same trusted Capture and Apply process that Classic did. There's almost no changes between the two except on how they communicate with their parent processes. And so the same logic that you use to pull data from Oracle or to apply data into Oracle is all the same.

10:25

Lois: And has the interface been upgraded as well?

Nick: We've added a really nice, easy to use web interface for the microservices version of GoldenGate. Not only is this web interface work with all your standard browsers, but it's also mobile friendly too. So I can actually control and administer GoldenGate right through my mobile device.

It also has new secure remote administration. This is something that the classic architecture was really missing. And so in the classic architecture, to use the command line interface, you had to log into the database server where GoldenGate was installed. Now, the command line interface, as well as the web interface and the REST API, all use remote administration and authentication. So that means that I can install the new command line interface or what we call admin client on my laptop locally and I can connect to any GoldenGate deployment as long as I have the username and password for that deployment. It's also more secure.

GoldenGate microservices can also be deployed on premise or in OCI as a service and now also on these third-party clouds like Azure and Google Cloud. And it's also easier for developers to integrate in with the APIs themselves. Everything that GoldenGate does through the admin client as well as the web UI can all be traced.

The REST API calls for GoldenGate are all fully published so you can get them right directly from the documentation, you can build your own web interface if you want to. So it makes it very easy.

The REST APIs are also streamlined. With a single REST API call, I can do something like add an Extract process, create it, set up my parameter file, and set up the trail files all with a single API command. Whereas in the past, it would require multiple command line interface commands to do that same thing. So it's extremely elegant, very advanced.

12:16

Nikita: What does the microservices architecture look like? I know it’s a bit complicated when we’re not actually looking at a diagram of it, but just a high level, can you explain the different parts of it?

Nick: It's pretty straightforward. But essentially what you've got on each system is a service manager. That service manager is then going to have a number of processes or services beneath it.

It'll have the configuration service that stores the checkpoint information for GoldenGate. It'll have the administrative service for the authentication and users, the distribution service to send the data across a network, a receiver service to receive that information, performance metrics to get the performance statistics out of GoldenGate. And then of course, you also have your Extracts and Replicats that capture and apply technology.

Each of those Extracts and Replicats will then connect to a database on the Extract side of things. That Extract is going to write to trail files. Those trail files are then going to be sent across the network where they're rebuilt on the target system and the Replicat’s going to consume them and apply them into the target database.

So the Replicat behaves almost like an end user. So it's taking that trail file data and simply converting it to DML operations, insert, update, delete, or a DDL operation in the case of Oracle, alter table, create table, et cetera, to go into that target database.  

13:39

Lois: To look at a diagram of this architecture and learn about it in more detail, check out the Oracle GoldenGate 23ai Fundamentals course on mylearn.oracle.com. So, Nick, if I’m looking to deploy GoldenGate, what should I primarily keep in mind?

Nick: So as you go to install GoldenGate and you look at a deployment, there's a couple of important environment variables that you want to make sure you're aware of. So one of the first ones is your OGG_Home.

This environment variable is extremely important. This is the location of the GoldenGate software itself. And I want to stress how important it is to always use version numbers when you're setting up your GoldenGate home.

When you go to install the software, if you're installing GoldenGate 23.5, use 23.5 within the home directory structure. If you're installing GoldenGate 23.7, use 23.7 inside that directory structure.

14:33

Nikita: Right… that way I’ll always know which versions are which, and it’ll make it really easy to upgrade and move from one version to the next. Ok, got it. What else, Nick?

Nick: There's a couple other important directories. You have your OGG_ETC_HOME. This is where things like the configuration files are going to reside, parameter files, all your certificates for security, including the wallets where we store the credentials for not only the database accounts, but also for the GoldenGate user accounts as well.

We have our GoldenGate variable home directory or VAR home. This is where all the GoldenGate log files are residing. And these are the log files that allow you to see what's going on in GoldenGate for auditing purposes. Anytime anybody makes a change to GoldenGate, you're going to see information go into the log files on what was happening and how it was working and what they did, what time they did, what command they issued.

Another big important feature about these log files is it also gives you error information and troubleshooting details. So if you ever need to find out what happened in GoldenGate, what went wrong, you would look at these log files to find out that information.

And then you also have your OGG_DATA_HOME. This is where those trail files are going to go. Essentially, this is kind of the queuing or overflow for GoldenGate.

There's a couple of other additional components. We've got the admin client. This is our command line utility. If you don't want to use a web browser or prefer a command line utility, you can use the admin client. The admin client is also fully scriptable.

So if you wanted to write scripts that would go off and automate things in GoldenGate, you can do that. A lot of customers did that with GGSCI in the classic architecture. You can do the same thing now with the admin client.

The other component is the microservices security authentication and authorization services. These handle communication security, especially making sure that any passwords or usernames and everything like that is all encrypted. And instead of using an actual username and password, everything through the product is going to be done through an alias. And then it also handles all the authorization authentication, permissions, user accountability, and roles within GoldenGate.

16:39

Lois: Anything else you’d like to talk about before we wrap up for today, Nick?

Nick: I also wanted to take a minute to talk about the REST API. All the microservices provide REST APIs to administer them and all of these are fully documented. They can be used by any client that can make REST API calls. So if you wanted to use Python, cURL, a web browser, you can do that as well. They're all just HTTP or HTTPS calls, get, put, patch, the standard REST API standards.

And then GoldenGate does provide our admin client as well as a WebUI that use these REST APIs under the covers if you ever wanted to get a more advanced look at how it works.

17:18

Nikita: Well, that’s all the time we have for today. Thanks for joining us, Nick. 

Lois: Yes, thanks Nick. We look forward to having you back next week to talk with us about security strategies and data recovery.

Nikita: And if you want to learn more about the topics we discussed today, head over to mylearn.oracle.com and take a look at the Oracle GoldenGate 23ai Fundamentals course. Until next time, this is Nikita Abraham…

Lois: And Lois Houston, signing off! 

17:43

That’s all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We’d also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.