In today's data-driven world, businesses rely heavily on robust data management solutions to extract actionable insights and drive strategic decision-making. Legacy systems, such as Informatica's PowerCenter, Metadata Manager, and others, have served organizations well over the years. However, the time has come to embrace the next generation of data management platforms, such as Informatica's Intelligent Data Management Cloud (IDMC). This blog post explores why current legacy Informatica customers should consider the transition and highlights the benefits of adopting IDMC. Comprehensive Solution Suite: Informatica's IDMC combines a comprehensive suite of data management products under one unified platform. With IDMC, customers gain access to a range of powerful tools, including PowerExchange (PWX), Data Quality (DQ), Data Transformation (B2B DT), Big Data Management (BDM), Data Integration HUB (DIH), and B2B Data Exchange (DX). This integration eliminates the need to purchase these products separately, streamlining the overall data management ecosystem. Enhanced Scalability and Flexibility: As organizations grow and their data management needs evolve, scalability becomes paramount. IDMC offers the scalability required to handle the ever-increasing volumes of data generated by modern businesses. Moreover, its cloud-based architecture allows for elastic scaling, enabling organizations to scale up or down based on demand. This flexibility ensures that businesses can adapt to changing requirements efficiently and cost-effectively. Next-Level Data Quality and Governance: Maintaining data quality and ensuring regulatory compliance is critical for any organization. IDMC leverages Informatica's advanced data quality and governance capabilities, empowering businesses to cleanse, standardize, and enrich their data easily. The platform's comprehensive data governance features, built-in metadata management, and business glossary capabilities enable organizations to establish a solid foundation for accurate and trusted data. Streamlined Operations and Simplified Management: One of the primary advantages of IDMC is its unified and user-friendly interface, allowing for simplified data management and operations. Organizations can leverage a centralized platform to design, develop, deploy, and monitor their data integration workflows and transformations. IDMC's streamlined management and monitoring capabilities lead to increased productivity and operational efficiency. Harnessing the Power of Big Data: In the era of big data, organizations must effectively handle vast amounts of diverse data sources. IDMC's Big Data Management (BDM) capabilities empower businesses to leverage the full potential of big data by seamlessly integrating and processing data from various sources. With IDMC, organizations can unlock insights from structured, semi-structured, and unstructured data, enabling them to make data-driven decisions confidently. Conclusion: In today's rapidly evolving data landscape, legacy data management systems can become a bottleneck for organizations striving to harness the power of their data. By migrating to Informatica's Intelligent Data Management Cloud (IDMC), current legacy Informatica customers can unlock many benefits. With its comprehensive suite of data management products, enhanced scalability, robust data quality and governance features, simplified management interface, and the ability to harness the power of big data, IDMC is poised to drive organizations toward a future of optimized data management and improved business outcomes. So, suppose you are still relying on legacy Informatica solutions. In that case, it's time to consider making the leap to IDMC and embark on a transformative journey toward intelligent and comprehensive data management such as IDMC.

Implementing Informatica B2B Gateway for EDI requires several steps and specialized skills and expertise with the technology. Here is a general high-level overview of the process: Define the requirements: Determine the business requirements for your EDI implementation. This includes identifying the types of documents you will exchange, the trading partners you will work with, and the specific data elements that must be included in each document. Configure the B2B Gateway: Install and configure the Informatica B2B Gateway software according to the specifications of your environment. This includes configuring the communication protocols, security settings, and other system parameters. Create and map EDI documents: This is where the majority of heavy-lifting in development is done. Use the B2B Gateway tools to create the EDI documents you will be exchanging with your trading partners. You will need to build mappings to map the data elements from your internal systems to the EDI document format and vice versa. Test the EDI transactions: Work with your trading partners to test the EDI transactions. This includes sending and receiving test documents, validating the data, and resolving any issues that arise. Deploy the system: Once testing is complete, you can deploy the B2B Gateway system for production use. Monitor and maintain the system: Monitor the system to ensure that it is operating correctly and resolve any issues that arise. You may need to update the configuration or make changes to the mapping as your business requirements change.

Electronic Data Interchange, or EDI, is the transfer of business documents electronically between two trading partners in a standardized format. This process eliminates the need for paper-based documents and manual entry of data, reducing errors, and improving efficiency. EDI has become more widespread with the growth of the internet and the need for companies to streamline their supply chains. EDI can be used for a variety of business documents, such as purchase orders, invoices, and shipping notices. The benefits of using EDI include faster processing times, improved accuracy, and reduced costs. By eliminating paper-based documents and manual data entry, companies can process transactions more quickly and with fewer errors. This can lead to faster turnaround times, improved customer service, and increased productivity. In addition to these benefits, EDI can also help companies improve their relationships with trading partners. By using a standardized format, companies can ensure that their trading partners receive accurate and complete information, which can help to build trust and reduce the risk of disputes. EDI is not without its challenges, however. Setting up an EDI system can be complex and time-consuming, and there can be issues with compatibility between different systems. In addition, there may be resistance to change from employees who are used to working with paper-based documents. Despite these challenges, many companies have found that the benefits of using EDI outweigh the costs. If you’re interested in learning more about EDI, please reach out as we extensive experience implementing EDI solutions in various industries such as healthcare, manufacturing, retail, and more using partnered technology products such as Informatica.

Talend CDC (Change Data Capture) es una herramienta que realiza un proceso de integración de datos, que replica y además sincroniza datos desde una fuente a diferentes destinos en tiempo real. Muchas veces se requiere que mantengamos nuestras bases de datos sincronizadas y son diferentes los casos de uso que pueden ser solucionados con CDC como por ejemplo: Tener nuestro DWH alineado con los últimos datos de nuestras fuentes transaccionales. Nuestra arquitectura híbrida necesita mantener el ambiente onpremise alineado con entorno cloud. Tenemos un sistema de replicación que consume mucho rendimiento, en el que necesitamos realizar consultas complejas sobre algunas columnas específicas de nuestros datos para saber si es un nuevo registro, si ha sufrido cambios o fue eliminado. Estamos usando nuestra base de datos transaccional como fuente de reportes, y por ende afectamos su rendimiento. Le gustaría tener un backup online. Quiere una base de datos distribuida. Entre otros. CDC puede ser usado en diferentes ambientes: onpremise, híbrido o cloud, y cuenta con conectores a los diferentes proveedores cloud como son Snowflake, AWS, o Azure, y servicios gestionados como Kafka, AWS kinesis o Azure Eventhub; adicionalmente cuenta con conectores y soporte onpremise como IBM(AS400), que es usado frecuentemente por entidades financieras y lo hace único en el mercado. ¿Cómo funciona? CDC identifica los diferentes cambios que hayan en la fuente y los replica en los destinos, estos datos replicados provienen de las diferentes operaciones realizadas como: INSERT DELETE UPDATE Las fuentes de datos que están produciendo cambios, tienen la capacidad de almacenar logs con los eventos de transacciones que se han realizado sobre los datos, estos logs pueden ser en diferentes formatos, pues varían de acuerdo al fabricante. El agente CDC actúa monitoreando y recolectando esos eventos transaccionales una vez, para almacenarlos en sus propios archivos de logs (Journal), esto, en caso de que sea necesario reusarlos en diferentes Jobs de replicación o de alguna otra manera, sin embargo, este proceso es ajeno a los recursos del sistema y no interactúa con ellos. En los casos en los que ya tengamos una base de datos creada y poblada como fuente, necesitaremos hacer solo una replicación completa, ya que de ese punto hacia adelante Talend CDC capturará los cambios realizados. Es posible aplicar scripts a los datos en la fuente para seleccionarlos, transformarlos o realizar agregaciones como también aplicar cambios en el destino previo a la integración. Su configuración es realizada en una consola gráfica en la que seleccionamos nuestra fuente y destino, esta opera con diferentes roles como: Administrador Operador Espectador Las réplicas son creadas en un ambiente y allí se llamarán modelos, un modelo contiene las fuentes y destinos; una vez configurados, podemos agregar opciones adicionales para estos, como por ejemplo, programar las ejecuciones, cambiar el framework con el que hacemos la integración, o supervisar el job a través de logs dentro de la herramienta, o fuera de ella en archivos de log. Existen diferentes frameworks con los que CDC puede hacer replicaciones de datos, cada uno de ellos cuenta con capacidades adicionales como agregar columnas en el destino, brindar transformaciones, agregar identificadores de secuencias, o incluso para trabajar con destinos de big data. Demo de una réplica Estándar Configuraciones Iniciales En este blog enseñaré una réplica estándar, por lo que seleccionaremos como framework “Free”, lo que significa que no se agregarán columnas o datos adicionales a los que ya se encuentran en la fuente, estaré usando la versión de Talend Change Data Capture V 7.15.0 Para este ejemplo ya tengo creadas las bases de datos fuente y destino en SQL Server a la cuales nombré “training_source” y “training_target”, y en la fuente tengo una tabla llamada “students” mientras que en el destino no he creado nada aún. Así mismo, la configuración de Talend CDC hacia la fuente y los diferentes componentes necesarios para el correcto funcionamiento de la herramienta ya se encuentran instalados, estos son: Motor de Captura Motor de Fuente Motor de destino Gestor de Talend CDC Conector ODBC 32 bits para SQL Server en el DSN (Data Source Administrator) SQL Server Comencemos: Iniciamos configurando la fuente en el menú File -> Source subscribe y diligenciaremos los datos de acuerdo con nuestra configuración de la fuente: Name (Alias): El nombre que quiera IP address: Es la ubicación del motor de origen de CDC donde se definió la conexión de la instancia de base de datos, esta configuración se realizó durante la instalación y configuración de CDC Source type: El motor de la BD fuente Instance or Server name: El que esté configurado en tu servidor Luego notaremos como en la fuente nos mostrará “TrainingSource” con un botón verde que nos servirá para conectarnos a esta: Una vez nos conectamos y veremos cómo se muestran los diferentes ambientes configurados donde podremos crear los modelos, para este ejemplo usaremos Training como nuestro ambiente: Ahora configuramos el journal, por lo que vamos al menú Source -> Journal Management Presionar clic derecho sobre la DB creada training_source y elegimos la opción “Start DB logging process” Luego podremos seleccionar un directorio en el que queramos almacenar el journal y presionamos Ok. Aún en el Journal Management hay que seleccionar la tabla que queremos monitorear y clic derecho sobre esta y presionar la opción “Start table logging process for”: Nota: Debe existir una llave primaria configurada en la tabla para iniciar el proceso de monitoreo sobre esta Después de iniciar el proceso de monitoreo sobre la tabla “students” necesitamos verificar que el journal está funcionando, para esto vamos a hacer un leve cambio sobre alguno de los registros de esta tabla: Nota: Hemos cambiado la ciudad John Doe de New York a Miami Ahora nuevamente desde el Journal Management clic derecho sobre “training_source” y “Display Journal”: Veremos el único receiver disponible por el cambio que acabamos de realizar: Lo seleccionamos y vamos a la pestaña “Posts” y desde allí veremos los cambios que se realizaron: Nota: Se observa el cambio de ciudad, el primer registro es “New York” y el segundo es “Miami” lo que nos garantiza el correcto funcionamiento del journal. Crear Réplica: El primer paso es crear un espacio de configuración dentro de un ambiente, cada réplica dentro de un mismo ambiente comparte los mismos parámetros que las demás y pueden ser administradas juntas. En la pestaña “Environment” clic sobre “Add” y diligenciamos los campos como el siguiente ejemplo: Environment: Código único del ambiente que se creará Source code: Código único para la fuente, debe ser de 3 caracteres Environment type: Para este ejemplo es “Training” Description: Se recomienda agregar descripción Clic sobre “Add” y tendremos el ambiente creado: Nota: El ambiente contiene una una pestaña para fuente donde se crea el modelo y otra pestaña del destino, adicionalmente tres pestañas donde se mostrarán propiedades de la replicación, y elementos gráficos de la réplica para la visualización del proceso. Me aseguro de estar en la pestaña “Source” y presiono clic derecho sobre “Models” y luego en “Add”: Y configurar el modelo de la siguiente manera: Nota: Se observan dos pestañas adicionales “Table Options” y “Script” a las cuales dejaremos por default. Model: Un código único para el modelo Description: Descripción del modelo Type: Para esta replicación usamos JOURNAL, sin embargo existe EXTRACT que permite una extracción full Clic en “Add” y el modelo estará disponible dentro del folder “Models” en la pestaña “Source” del “Environment”: El siguiente paso es agregar Tablas al modelo, expandiendo el Modelo en el botón [+], luego clic derecho sobre “Tables” y finalmente clic en “Add” Una ventana aparecerá para seleccionar las tablas que agregaremos al modelo: Nota: Como se puede observar, estamos usando el framework “Free” que no genera cambios adicionales a nuestros datos Con el botón “Query” podremos listar todas las tablas disponibles Seleccionar las tablas, en este caso “students” Mover la tabla seleccionada al modelo Finalizar La tabla aparecerá creada dentro de nuestro modelo: El siguiente paso es crear el destino, desde la pestaña “Target” de nuestro ambiente damos clic derecho en “Targets” y luego “Add”: Se abrirá la ventana para el nuevo Target: Target: Código único para el destino Description: Descripción del destino Target type: valor por default Name: Nombre de la instancia de nuestra base de datos destino Address: dirección del servidor donde está la base de datos destino Exclude: se llena automáticamente en cuanto ponemos el nombre de la instancia Entonces clic en “Add” y veremos nuestro target creado: El siguiente paso es crear una distribución que significa asociar nuestro modelo a un destino, lo podemos realizar desde la pestaña “Source” o “Target”; expandir nuestro target creado y clic sobre “Distribute”: Y movemos el Modelo desde el panel izquierdo al derecho y clic sobre “Ok”: Una ventana se abrirá automáticamente para finalizar la configuración de la distribución: Seleccionar el tipo de base de datos Doble clic sobre el conector Se abrirá una ventana para ingresar los datos de nuestra base de datos de destino Clic en “Ok” para agregarla Si la conexión es correcta, una ventana de notificación nos mostrará “Connection Successful” y nos mostrará las bases de datos disponibles allí, y finalmente clic en “Add”: Podemos ver ahora desde la pestaña “Map” cómo el target T01 está asociado al modelo M01: El siguiente paso es crear la tabla destino, entonces desde la pestaña “Target” damos clic en el modelo “M01” luego seleccionar la tabla a replicar, clic derecho sobre esta y finalmente “Create Target Table”: Luego de esto veremos que una nueva ventana aparece, mostrando el DDL de la nueva tabla usando la instancia de la base de datos de destino: Clic en “Execute” y luego chequeamos en la base de datos de destino para evidenciar que la tabla fue creada allí, pero se encuentra vacía: Ejecutar Réplica: Ahora que ya está creado el target, el modelo y la distribución, debemos ejecutar la réplica; desde la pestaña “Map” clic derecho sobre el Modelo M01 y luego clic en “Properties”: Desde la pestaña “Recovery” seleccionamos el check box “Load”: Luego vamos a la pestaña “Activity”, una ventana de confirmación nos pedirá si deseamos recargar todas las tablas en esa distribución, a lo que diremos “Yes”. Ahora desde la pestaña “Activity” clic sobre el botón “Start”: Después de iniciar la replicación, los campos de la sección “Counters” y “Last operation” nos mostrarán los resultados y así mismo la distribución quedará activa y a la espera de algún cambio que ocurra en la fuente para replicarlo automáticamente a menos de que deseemos detenerla: Nota: Como se observa, tenía 3 registros en la fuente y esos mismos fueron seleccionados, enviados y agregados. Cuando el modelo esté activo, será de color verde, de lo contrario será amarillo, o puede ser rojo si se encuentran errores durante la ejecución: Ahora se debe revisar en la tabla destino para evidenciar que los registros fueron replicados: Finalmente detenemos la replicación dando clic derecho sobre el modelo que aparece en verde y clic en “Stop” y de esa manera finalizamos la réplica estándar. En resumen: Talend CDC nos permite diferentes modos y ofrece muchas más características para la replicación en tiempo real; en el ejemplo anterior realizamos una replicación simple con los mismos campos en fuente y destino dentro del mismo servidor, así como una ejecución manual dado que pueden ser también programadas. En los próximos posts, enseñaré otras características o funciones con las que Talend CDC trabaja, además de evidenciar las diferencias con otros frameworks allí disponibles. Fredy Antonio Espitia Castillo Talend Developer Certified https://www.linkedin.com/in/fredy199601/

Talend CDC (Change Data Capture) is a tool that performs a data integration process that replicates and syncs data from one source to different targets in real-time. In a lot of business scenarios, it is required to keep databases synchronized, and there are many of those use cases that can be solved with Talend CDC, such as: Keeping the DWH aligned with the latest data from transactional sources Hybrid architecture needs to keep the cloud environment aligned with the on-premises data environment A very performance-consuming replication system that needs to perform complex queries on specific columns of our data to find out if it is a new, changed, or deleted record. A transactional database is used as a reporting source, thus affecting its performance. Having an online backup Having a distributed database CDC can be used in different environments: on-premises, hybrid, or cloud, and it has connectors to different cloud providers such as Snowflake, AWS, or Azure, and managed services such as Kafka, AWS kinesis or Azure Eventhub. It also has connectors and support for on-premises servers such as IBM AS400, which is frequently used by financial entities and makes it unique in the market. How does Talend CDC work? CDC identifies the different changes that are in the source and replicates them in the target source. The replicated data comes from the different operations carried out such as: INSERT DELETE UPDATE The data sources that are producing changes can store logs with the transaction events that have been carried out on the data. These logs can be in different formats, varying according to the manufacturer. The CDC agent works by monitoring and collecting those transactional events once, to store them in its own log files (Journal), in case it’s necessary to reuse them in different replication. However, this process is oblivious to system resources and does not interact with them. In cases where we already have a database created and populated as a source, we will need to only do one full replication. From that point forward, Talend CDC will capture the changes made. It’s possible to apply scripts to the data in the source to select, transform or perform aggregations as well as apply changes to the target prior to integration. Its configuration is done in a graphical console in which we select our source and target. It operates with different roles as: Administrator Operator Viewer The replicas are created in an environment, and there will be called models; a model contains the sources and targets; Once configured, we can add additional options for these, such as scheduling the executions, changing the framework with which we integrate, or supervising the job through logs in the tool or log files. There are different frameworks with which CDC can do data replications, and each of them has additional capabilities, such as adding columns in the target, providing transformations, adding sequence identifiers, or even working with big data targets. Demo of a Standard Replication Initial Settings In this blog, I will show a standard replica, so we will select the framework as “Free”, which means that no additional columns or data will be added to what is already in the source; I will be using Talend Change Data Capture version V 7.15 .0. For this example, I have already created the source and target databases in SQL Server, which I named "training_source" and "training_target" respectively; in the source, I have a table called "students" while in the target, I have not created anything yet. Likewise, the configuration of Talend CDC towards the source and the different components necessary for the correct functioning of the tool are already installed, these are: Capture Engine Source Engine Target Engine Talend CDC Manager 32-bit ODBC connector for SQL Server in the DSN (Data Source Administrator) SQL Server Let's start: We start by configuring the source in the File -> Source subscribe menu and fill in the data according to our source configuration: Name (Alias): The name that you prefer IP address: Is the location of CDC source engine where the DB instance connection has been defined, this setup was done during the CDC installation and configuration Source type: The engine of the source DB Instance or Server name: The one configured on your server Then we notice that in the source it will show us "Training Source" with a green button that will help us to connect to it: When we connect to that source, we will see that the different configured environments where we can create the models are shown, for this example, we will use Training as our environment: Now we configure the journal, so we go to Source -> Journal Management Right click on the created DB "training_source" and choose the option “Start DB logging process” Then we can select the directory in which we want to store the journal and press Ok. Still in Journal Management, we have to select the table we want to monitor and right click on it and then, press the option “Start table logging process for”: Note: A primary key must be configured in the table to start the monitoring process on this. After starting the monitoring process on the "students" table, we need to verify that the journal is working, for this we are going to make a slight change on one of the records on this table: Note: We have changed John Doe's city to Miami instead of New York Now again in the Journal Management right click on “training_source” and “Display Journal”: We will see the only receiver available, which is the one written for the change we just made: We select it and go to the "Posts" tab and from there we will see the changes that were made: Note: The change of city is shown, the first record is "New York" and the second is "Miami" which guarantees the correct functioning of the journal. Creating the Replication: The first step is to create a configuration space within an environment; each replica within the same environment shares the same parameters and can be managed together. In the "Environment" tab, click on "Add" and fill in the fields like the following example: Environment: unique code of the environment to be created Source code: Unique code for the source, must be 3 characters exactly Environment type: For this example it’s “Training” Description: It’s recommended to add a description Click on “Add” and we will have the environment created: Note: The environment contains a tab for the source where the model is created and another tab for the target. Additionally, three tabs where the replication properties will be displayed, and graphic elements of the replica to visualize the process. Now from the “Source” tab right click on “Models” and then on “Add”: And configure the model as follows: Note: Two additional tabs “Table Options” and “Script” are shown, for which we will use the configuration set by default. Model: A unique code for the model Description: Description of the model Type: For this replication, we use JOURNAL, however, there is EXTRACT that allows a full extraction Click on “Add” and the model will be available in the “Models” folder in the “Source” tab of the “Environment”: The next step is to add Tables to the model, expanding the Model on the [+] button, then right-clicking on “Tables” and finally clicking on “Add” A window will appear to select the tables that we will add to the model: Note: As you can see, we are using the "Free" framework that doesn’t generate additional changes to our data With the "Query" button we can list all the available tables Select the tables, in this case "students" Move the selected table to the model Finish The table will appear created inside our model: The next step is to create the target, from the "Target" tab of our environment we right click on "Targets" and then "Add": The window for the new Target will open: Target: unique code for the target Description: Description of the target Target type: default value Name: Name of the instance of our target database Address: address of the server where the target database is located Exclude: it’s filled automatically as soon as we put the name of the instance Then click on “Add”, and we will see our created target: The next step is to create a distribution which means associating our model to a target, we can do it from the "Source" or "Target" tab; expand our created target or source, and click on “Distribute”: And we move the Model from the left panel to the right and click on “Ok”: A window opens automatically to finish the configuration of the distribution: Select the type of database Double-click on the connector A window will open to enter the data from our target database Click on “Ok” to add it If the connection is correct, a notification window will show us "Connection Successful" and will show us the databases available there, and finally click on "Add": We can now see from the "Map" tab that the target T01 is associated with the model M01: The next step is to create the target table, then from the "Target" tab we click on the "M01" model, then select the table to replicate, right-click on it, and finally "Create Target Table": After this we will see that a new window appears showing the DDL of the new table using the target database: Click on “Execute” and then we check in the target database to show that the table was created there, but it is empty: Run Replication: Now that the target, the model and the distribution are created, we must execute the replication; from the “Map” tab, right click on Model M01 and then click on “Properties”: From the "Recovery" tab we select the "Load" check box Then we go to the "Activity" tab, a confirmation window will ask us if we want to reload all the tables in that distribution, to which we will say "Yes". Now from the "Activity" tab click on the "Start" button After starting the replication, the fields of the "Counters" and "Last operation" section will show us the results and likewise, the distribution will remain active and waiting for any change that occurs in the source to replicate it automatically: Note: As shown, we had 3 records in the source and those same ones were selected, sent and added to the target. When the model is active, it will be green, otherwise it will be yellow, or it can be red if errors are encountered during execution: Now it must be checked in the target table to show that the records were replicated: Finally, we stop the replication by right-clicking on the model that appears in green and clicking on “Stop”. Talend CDC offers many more features for real-time replication. In the previous example, we perform a simple replication with the same source and target fields, as well as a manual execution since they can also be scheduled. In the next posts, I will show other features or functions that Talend CDC works with, as well as highlighting the differences with other frameworks available there. Fredy Antonio Espitia Castillo Talend Developer Certified https://www.linkedin.com/in/fredy199601/

Last week on March 9, 2023, I attended the NYC Snowflake Data for Breakfast event. The event was packed with attendees from various industries and featured a lineup of great speakers who shared their experiences and insights on using Snowflake's platform. Also, to make things better, Snowflake had some great food and coffee flowing, which was needed since the event started at 8 am EST. Now the event kicked off with Brent Bateman, a Principal Sales Engineer at Snowflake, who discussed the capabilities of Snowflake, with a focus on its self-managed automated administration and unified governance platform. He highlighted how Snowflake is helping companies easily manage and access their data with less complexity and more agility. A lot of us know this today, but it was a good way to get the room to wake up and excited for the morning event. One thing Brent did not discuss though was the recent acquisition with Streamlit, but we all know there will be more to come on this in the future with Snowflake. Next, Jonathan Hyman, Co-Founder & CTO at Braze, shared how they use Snowflake to gain better insights into their customers and build more personalized experiences. He emphasized the importance of data-driven decision-making in today's business landscape and how Snowflake is an essential tool for achieving that goal. In addition, it was a breath of fresh air as Jonathan also walked us through at a high-level his current architecture and their customer journey. The event also featured a panel discussion with Jai Subrahmanyam, Senior Vice President, Head of Data Governance at The Blackstone Group, and Phil Andriyevsky, Partner- Wealth and Asset Management at EY. They shared how Snowflake has helped The Blackstone Group in leveraging data for better decision-making and driving business outcomes. They also discussed the value Snowflake has brought to their respective organizations in terms of scalability, performance, and cost savings. The event concluded with all speakers coming together to answer questions from the audience. Attendees had the opportunity to learn from Snowflake experts and industry leaders and gained valuable insights on how Snowflake can help their organizations effectively manage and utilize their data. Overall, this was a great event and extremely informative and insightful. Also, I look forward to exploring Snowflake capabilities and how they are being used with their current customers. Also, thanks again to the Snowflake Partner team for inviting Pingahla Colombia and Pingahla NorAm team to all the events being hosted around the world! Next up Snowflake Summit on June 26th! I can't wait to be in attendance.

As this week draws the Data for Breakfast events with our Snowflake partner. I thought it would be only right to put together a blog post on the latest news about Snowflake's Streamlit acquisition. When I saw this acquisition, I thought this was a great job by Snowflake! But I’m still waiting on Snowflake to purchase dbt...😊 Now some of you may not have heard about Streamlit. Now for those who have only heard about Steamlit now, let me share with you a quick blurb about the company that you can easily find on the web. Streamlit is an open-source framework that allows data scientists and machine learning engineers to build web applications for their data projects. With Streamlit, users can create interactive and customizable web interfaces for their machine-learning models, data visualizations, and data exploration tools without needing to know how to write front-end web code. Now, what is the value of Streamlit to organizations and companies like yours? Well, their software allows the ability to simplify and streamline the development process. From what I have read and seen on YouTube. Using Streamlit, companies can build and deploy interactive applications faster and with fewer resources than with traditional dev methods. Now the kicker for the solution's value is that it can lead to faster time-to-market for new products, and solutions, improved collaboration between data engineers, data scientists, and business users, and greater agility in responding to changing market needs. To me, this is a game-changer! Now if it can do all of this why would Snowflake would want to purchase Streamlit? Now I do not sit on the Snowflake board, work for the company, etc. I have very little insight on the WHY, but being a technologist, I would believe the reason Snowflake purchased Streamlit is due to many of the benefits I highlighted above which include; Allowing Snowflake customers to quickly build and deploy web applications on top of their data. This could help Snowflake differentiate its offering from competitors and provide additional value to its customers. Snowflake's expertise in data management and analytics could help streamline Streamlit's development process and provide access to a larger customer base. This could help accelerate Streamlit's growth and adoption in the market. The acquisition could potentially lead to synergies in research and development, with both companies sharing knowledge and expertise in the areas of data management, analytics, and web application development Either way, this is a win, win for Snowflake customers and partners like Pingahla. I am excited to be attending the upcoming NYC Snowflake Data for Breakfast and I hope this is a topic that will be discussed. Snowflake + Streamlit News: https://investors.snowflake.com/news/news-details/2022/Snowflake-Announces-Intent-to-Acquire-Streamlit-to-Empower-Developers-and-Data-Scientists-to-Mobilize-the-Worlds-Data/default.aspx

In today's data-driven world, data has become one of the most valuable assets for businesses. However, the increasing amount of data being generated and processed also poses significant risks to data security, privacy and data regulations. Therefore, it has become essential for businesses to implement robust data governance and data security practices to mitigate these risks. Now I understand, not all organizations have a clear data governance strategy or org in place, but I am seeing more and more of my customers make this investment due to data regulations such as CCPA, GDPR, and Virginia CDPA to name a few. However, business and development teams often face significant challenges while working with data governance and data security teams. These teams can slow down the development process, making it difficult for business and development teams to deliver new features and functionality on time. This delay can cause frustration for the business users, developers and may impact the overall success of the project. Despite these challenges, it's crucial for the business and development teams to work closely with data governance and data security teams. These teams are responsible for implementing the policies, procedures, and technologies that mitigate security risks, ensure data compliance, and protect sensitive information. Here are some reasons why: Mitigating security risks: Data governance and data security teams are responsible for identifying and mitigating security risks. These teams establish security policies and procedures, perform security assessments, and implement security controls to protect data from unauthorized access, theft, or misuse. By working closely with these teams, developers can ensure that the applications they develop are designed and implemented with security in mind, reducing the risk of security breaches. Ensuring data compliance: Data governance and data security teams are also responsible for ensuring that the organization's data practices comply with relevant laws and regulations, such as CCPA, Virginia CDPA, GDPR or HIPAA. These regulations have strict requirements for data protection, data privacy, and data access, which can be challenging for development teams to navigate. By working closely with data governance and data security teams, developers can ensure that their applications comply with these regulations, reducing the risk of non-compliance and potential legal and financial consequences. Protecting sensitive information: Data governance and data security teams are responsible for protecting sensitive information, such as personal information, financial information, and intellectual property. By working closely with these teams, developers can ensure that the applications they develop are designed and implemented with data protection in mind, reducing the risk of sensitive information being exposed or misused. Now, let's take a look at some examples of companies that have faced consequences for poor data governance or data security practices: Facebook: In 2018, Facebook was embroiled in a massive data privacy scandal involving the data analytics firm Cambridge Analytica. It was discovered that Cambridge Analytica had obtained data from millions of Facebook users without their consent, which was used to influence political campaigns. This scandal led to a massive loss of trust in Facebook, and the company faced intense scrutiny from regulators and lawmakers around the world. Equifax: In 2017, credit reporting agency Equifax suffered a massive data breach that exposed the personal information of approximately 143 million people. The breach was caused by a vulnerability in Equifax's web application framework, which had not been patched despite a security alert being issued months earlier. The company faced intense criticism for its poor data security practices, and it ultimately had to pay a $700 million settlement to consumers, regulators, and lawmakers. Citibank: In 2020, Citibank was fined over $400M due to risk management and data governance issues. The bank had failed to implement proper risk management practices and internal controls, which led to a massive error in its payments system. The error resulted in Citibank accidentally transferring nearly $900 million to a group of creditors of Revlon, the cosmetic company. Citibank was unable to retrieve the funds, and it ultimately had to absorb the loss. The incident highlighted the importance of robust risk management and data governance practices, as failure to implement these practices can result in significant financial losses and regulatory consequences. These examples illustrates how poor data governance and risk management practices can have severe consequences for organizations like these three examples. It's essential for business and development teams to work closely with data governance and security teams to ensure that the applications and solutions they develop are designed and implemented with these practices in mind. By doing so, organizations can reduce the risk of costly errors, data breaches, and regulatory fines. In conclusion, while working with data governance and data security teams can be challenging, it's essential for development teams to prioritize these practices. By doing so, business and developers can ensure that the applications they develop are designed and implemented with security and compliance in mind, reducing the risk of security breaches, non-compliance, and potential legal and financial consequences. The mind set that these two parts of the organization are different should not exist. Think of these two parts of the org as an extended part of your team wanting you on the business or delivery team to succeed while making sure the proper governance and security is in place!

At Pingahla we work with customers who are at different stages of their cloud journey. A common theme that we see is the selection of the right cloud data warehousing and analytics platform. This becomes especially important as the wrong choice may seriously hinder the speed and agility of digital transformation and the company's ability to respond to changing market dynamics. In this post we will evaluate and compare two major players in this space - Snowflake and DataBricks DataBricks and Snowflake are both cloud-based data warehousing and analytics platforms. While they share some similarities, there are also some key differences between the two. One major difference is that DataBricks is a unified analytics platform that combines data engineering, data science, and business analytics in a single environment, while Snowflake is a cloud-based data warehousing solution that focuses specifically on data storage and querying. Another difference is that DataBricks uses an in-memory computing model, which allows for fast processing of large amounts of data, while Snowflake uses a columnar storage model and separates compute and storage resources, which can make it more scalable and cost-effective for certain use cases. Additionally, DataBricks offers a range of integrated tools and services for data engineering, machine learning, and business intelligence, while Snowflake integrates with a variety of third-party tools and services for data integration, visualization, and analytics. Overall, the choice between DataBricks and Snowflake will depend on the specific needs and goals of an organization. DataBricks may be a better fit for organizations that need a full-featured analytics platform with integrated tools and services, while Snowflake may be a better fit for organizations that have specific data warehousing requirements and want a scalable and cost-effective solution. At Pingahla, we help solve complex data challenges for our customers. We are experts in building cloud based data warehousing and analytics solutions. To know more, please reach out at info@pingahla.com

Our instructional step by step video by our #Pingahla certified #Talend Kuldeep Singh will show you how to create an ETL data pipeline using Talend Studio and its #Amazon #S3 bucket connector. Below are the steps that will show you how to build out such a data pipeline, else you can scroll to the bottom of this blog post and check out our YouTube instructional video on the following topic. Open Talend Studio. Using the Local or select a Connection. In this example, we will select Local and select an existing project, "Local_Project." You can do the same or select any other project where you want to create the data pipeline. Once the Talend Studio has started, make sure you are in the Integration Perspective, if it is not already selected. You can also change the perspective by going to Window > Perspective > Integration. This can also be done from the taskbar by clicking on Integration as shown in the picture below: Create a new Job by right-clicking on Repository > Job Designs, then selecting Create Standard Job. Enter the Job Details and click on finish when done. Always put in a description for the job, this will help others to understand the purpose of the job in a collaborative environment. Now we will start adding the components from the Palette to the Job. For the purpose of the guide, we will generate data using the tRowGenerator component available in Talend Studio Palette. The generated data will be written in a delimited file. The generated file will then be placed on an S3 bucket. First, we will start creating a file with dummy data using the tRowGenerator component. In the Palette view, begin typing the name of the component, in this case, tRowGenerator, into the Find Component box and then click the Search button. You can also press the Enter key. You can also click anywhere in the Designer and start typing the name of the component to add it. Once, the component is added to the designer, double click the component to open it. Here we will define the structure of data to be generated. ·Add as many columns to your schema as needed, using the plus (+) button. Type in the names of the columns to be created in the Columns area and select the Key check box if you wish to define a Key for the generated data Make sure you define then the nature of the data contained in the column, by selecting the Type in the list. According to the type you select, the list of Functions offered will differ. This information is therefore compulsory. Once done, Click on OK to close the dialog box. Now to write the generated data by the tRowGenerator to a flat file on your local system, we will use the tFileOutputDelimited component. Just follow the same steps that we used to add the tRowGenerator component. Once added, your Job should look like this. Now we will connect the tRowGenerator component to the tFileOuputDelimited component. To do that, right click the tRowGenerator_1 component, then select Row > Main. Move the mouse over top of tFileOuputDelimited_1 component and click on it. Once done, your Job should look like this. Now let’s configure the tFileOutputDelimted component. To do that, double click the tFileOutputDelimted_1 component and you would notice that the Component View has opened at the bottom of the screen. Provide the absolute path of the file where do you want to output the generated data. In this example, it is "C:/Users/PINGAHLA/Desktop/Talend_Demo/demo.txt" Now let’s add the tS3Connection & tS3Put component to the Job. Now let’s connect the initial Subjob that we are using to create a file and connect it to the tS3Connection. Right-click on the tRowGenerator_1 component, then select Trigger > On Subjob Ok. Move the mouse over top of tS3Connection_1 component and click on it. Similarly, Right-click on the tS3Connection_1 component, then select Trigger > On Subjob Ok. Move the mouse over top of tS3Put_1 component and click on it. Your final Job should look like below. Now let’s configure the tS3Connection & tS3Put component. Double-click tS3Connection_1 to open its Basic settings view on the Component tab. In the Access Key and Secret Key fields, enter the authentication credentials required to access Amazon S3. Ensure that the values are enclosed in double-quotes. The tS3Connection component should have all the details as above. Double-click the tS3Put component to open its Basic settings view on the Component tab. Select the Use an existing connection check box to reuse the Amazon S3 connection information you have defined in the tS3Connection component. In the Bucket field, enter the name of the S3 bucket where the object needs to be uploaded. In this example, it is talend-data and the bucket is already present in Amazon S3. Note: Ensure that the bucket is already present in your Amazon S3 Instance. In the Key field, enter the key for the object to be uploaded. In this example, it is demo. Note: This field ensures the filename for the file that is being uploaded to the S3 bucket. In the File field, browse to or enter the path to the object to be uploaded. In this example, it is "C:/Users/PINGAHLA/Desktop/Talend_Demo/demo.txt" Press Ctrl + S to save the Job. Press F6 to run the Job. Run details on a successful run should look like below: Login into your S3 bucket and you will find that the file has been uploaded to your S3 Bucket. In case you want to upload the file to a particular folder in your S3 bucket, enter the entire path along with the filename in the Key field. In this example, it is “demo/demo.txt”. Press Ctrl + S to save the Job. Press F6 to run the Job. Login into your S3 bucket and you will find that the file has been uploaded to your S3 Bucket. Check out our instructional video on how to create an ETL data pipeline using Talend Studio and its #Amazon #S3 bucket connector

More and more of Pingaha's Talend customers are looking for ways for self-service around data extraction, transformation, and insights, allowing them to conduct their business in a globally competitive market. At Pingahla we are constantly educating our Talend customers on many of the current and new self-service Talend solutions. Within this blog posting, I will be discussing the high-level benefits of Talend's self-service solutions such as Stitch, Data Pipelines and Data Preparation. Let me first start with Stitch. In Nov 2018, Talend acquired Stitch to complement its unified platform, allowing its users to have the ability for self-service for data extraction for non-technical users. Stitch is a web-based cloud EL (Extract and Load) solution in which users can quickly extract data from a wide variety of supported sources and load that data into a specific supported target with Stitch. This can be down in as little as 7 major steps. 1. Simply go to www.stitchdata.com. 2. If you have an account either Sign In or Sign Up for a Stitch account. Note: Within these steps below, I have created my account. 2.1. Enter the required requested information to create your Stitch account. 2.2 Stitch will send an email to confirm your email address. Please click the confirmation email, as this will confirm your email address and account with Stitch. 3. Once you have either logged in or created your account, next you will want to select the data source in which you would like to extract the data. Stitch has around 108 data source connectors. Quickly find your source and select the connector. In this example, we have selected the "Microsoft SQL Server Integration" source connector. Note: If you cannot find your source connector, you have the ability to suggest a connector to the Stitch-Talend team. If you are a paid customer, most likely your suggestion will move hirer within the development queue for Stitch-Talend. 4. Next, you will need to enter in details of your data source and select the specific data tables so that Stitch can connect and extract the data. Note: You can also invite a member of your team to enter in the connection details if you do not have it. 5. Once you have entered your source details you will then need to select your target destination. Stitch has 8 target destination connectors in which you can choose from. In this example, we are selecting Amazon S3. 6. Once you have selected your target destination, you will then need to enter your target connection details. 7. Once you have entered in your target destination, you will set up your schedule and PRESTO, you have created your very first self-service data pipeline with Stitch. Another truly self-service ETL (Extract, Transform and Load) solution from Talend is Pipeline Designer. Talend's Pipeline Designer is a cloud-based ETL solution that recently was released to the public in April 2019. Not only does it allow for its users to build-out ETL data pipelines using drag and drop functionality, but it also allows users to incorporate and support Python coding! A huge yay for Data Scientist. Now there are a few restrictions around the Python competent which I recommend you review the Talend documentation. But here are a few steps to get you started with Talend's Pipeline Designer. 1. To get started with Pipeline Designer you will need to first have access to Talend Cloud. Note: Talend Cloud has made recent announcements with its latest update with Talend Cloud on Azure, so you will need to make sure you select the correct cloud provider and zone before logging in. In this example, I am logging into the Talend Cloud AWS, East Zone. 2. Once you have logged into Talend Cloud, you will be directed to Talend's Cloud portal which will be the "Welcome" page. Here you will be allowed to Launch the different Talend Cloud applications or watch specific tutorials on the different Talend Cloud applications. For the purpose of this blog posting, you will need to launch Pipeline Designer. There are two ways to launch Pipeline Designer, which include finding the application on the Welcome page and click on the Launch button, or within the left-hand corner select the "Select an App" drop-down and select Pipeline Designer. 3. Once you Launched Pipeliner Designer you will either be directed to Pipeliner Designer or a Pop Up that will request you to set up your Talend Pipeline Designer remote or cloud engine. In this blog posting, I will not be going through the steps in setting up a Talend remote/cloud engine for Pipeline designer as I will revisit this in a future post. 4. After launching Pipeline Designer you will be directed to the "Datasets" page within Pipeline Designer. You can create your data pipeline from the "Datasets" page or the "Pipelines" page. In my blog posting, I will be creating my data pipeline from the "Datasets" page by right-clicking the Customer data source and Add Pipeline. 5. Once you add your Dataset to a Pipeline or start creating a data pipeline from scratch, you will be directed to the Talend Pipeline Designer Canvas. Here you will be able to create your ETL or ELT data pipeline process. In this example, you will see we already have our Customer data source as the first component in our data pipeline. 6. Once you have established a pipeline and data source, you can then add an additional Talend processor competent for your data pipeline. To add a Pipeline processor, you will want to click the "+" button within the data pipeline flow. The data processor will allow you to transform your data. Talend offers 9 major processor components with Talend Pipeline Designer. For the full list please visit; Talend Cloud Pipeline Designer Processors Guide. 7. After adding the different Talend processor competent to your data pipeline, you will then need to add the Target destination. To add the Target destination click on the "Add Destination" button within the data pipeline flow. Note: You have the ability to preview the data as it flows through the different Talend processors by clicking on the "Preview" button within the data pipeline flow. 8. Now that you have completed building out your ETL/ELT Talend Data Pipeline Designer data pipeline, you will want to run your pipeline. To simply run your data pipeline, click on the "Play" button on the top middle right-hand corner. 9. If the data pipeline you have completed is correct. You will receive a green pop up notifying you your data pipeline has completed and loaded successfully or unsuccessfully. One of my favorite Talend self-service solutions is Data Preparation. Not only do I believe this is a favorite self-service solution from Talend, but many of my customers would say the same. Talend Data Preparation aka, Data Prep is a on-prem or cloud solution that is truly simple and easy to use solution that enables users to quickly identify data anomalies as well as speed up the delivery process of cleansed or enriched data by allowing non-technical users to collaborate with IT on collaborative pipeline solutions, or enrich data sets for BI reporting needs without involving IT. Some of the major benefits of Talend Data Preparation that make it my favorite solution is the following; * Collaboration in development with non-technical users * Replacement of STTMs (Source To Target Mappings) * Self-service to profile, transform, and enrich data sets These are the three major reasons why I enjoy using Talend Data Preparation. The simple and easy to use solution looks like Microsoft Excel. So off the bat, there is nothing technical or scary about the solution. But let me break it down. 1. Talend Data Preparation is an on-prem and Cloud-based solution. Here I will focus on the cloud version. The on-prem version works very similarly, except when connecting to data sources and how to collaborate with a developer's Talend Studio data pipeline. But I will get into that in a later post. 2. Similar to steps 1-3 outlined in the Talend Pipeline Designer steps, you will need to make sure you are logged into Talend Data Preparation. Or if you are using the on-prem solution, you will need to have the on-prem solution installed with a valid license. 3. Once you are logged in, you will need to make sure you have the data sources you would want to profile, transform or enrich. Talend allows users to add different types of datasets that include local flat files (txt, Excel, csv, and etc), talend jobs, databases, Amazon S3, and Salesforce. 4. After you have established and imported your datasets, you can now create a "Preparation" in Talend's simple and easy to use grid-like graphical interface. 5. To create a "Preparation," simply click on the "Add Preparation" button. The "Add Preparation" button will allow you to profile, enrich and transform your source data. 6. Now that you have imported and set up your connections to your Datasets, and have created your first preparation you will now notice your add has been uploaded to Talend's grid-like interface. Now what you may or may not notice off the bat is that Talend Data Preparation has quickly profiled the data for you highlighting its analysis. What do I mean by this? Now that your data has been created as a preparation, take a look at the following data highlights. 7. Column Discovery: Talend has analyzed the metadata doing its best to determine the column metadata. You have the option to update this manually if the analysis is incorrect. 8. Quality Bar: Talend Data Preparation, highlights columns in different colors (Green, Black, and Orange). The different colors are used to identify valid records, empty records, and invalid records. You can also create your own rules on the analysis. 9. Talend Data Preparation, allows users to enrich and transform the data set without enriching or transforming the data directly on the source. But within Talend Data Preparation you are able to leverage "FUNCTIONS" that allow you to add enrichment or transformation rules to the data set. 10. Data Profiling: As mentioned prior, another huge benefit of Talend Data Preparation is the data profiling aspect. Users are able to showcase and quickly profile the data columns to have a better understanding of the data. 11. Data Lookup: With an self-service solution you need to have the functionality to do a lookup for either validation or enrichment, and Talend Data Preparation allows you to easily do this with the lookup feature. 12. Filters: In addition, to adding lookups you can also add filters to the data set. This can quickly be done by selecting the Filters option on the top middle-right hand corner of Talend Data Preparation. 13. Recipe: Now here comes the best feature, "Recipe." Now as you enriched and transformed your data while doing this Talend Data Preparation was documenting each transformation step allowing you to no longer needing to create lengthy STTM's as well as see how the data will change when you apply a function. In addition, this "Recipe" can be shared with a developer's Talend ELT or ETL job in Talend Studio, allowing a user to collaborate with IT. 14. Exporting: The last beneficial feature is the exporting feature. Talend Data Preparation allows users to export the data in multiple formats that include flat files, Tableau and Amazon S3. Hopefully, you enjoyed this blog posting. Please leave a comment for any additional feedback or questions. Also, if you are interested in learning more about Pingahla's Talend services, please contact sales@pingahla.com.

If you are a Talend retail customer, you may have been holding off moving to Talend Cloud because it was hosted on AWS, or you may be an Azure customer looking for the native Azure solution. Well, now you have an option in using Talend Cloud on Microsoft Azure. Talend made this huge announcement in Oct 2019, and many on-prem retail customers want to flip the cloud switch. Hopefully, you had the opportunity to check out Talend's Webinar on Oct 30, 2019, that introduced the Talend Cloud Azure offering, but if you missed it you are in luck as Talend is having another live Webinar on Nov 13, 2019, 11 am SGT. But before making that jump to Talend Cloud on Azure, I put together this blog post to help educate you on Talend's Azure offering as it is a little different than the Talend Cloud AWS offering due to the Talend Azure Roadmap, which I will address within this blog posting. Now with Talend Cloud rolled out on Microsoft Azure, you will have to note for this first phase for Talend you will only be able to leverage Azure on the United States - West Zone. Now, why do I bring this up? If you are a current Talend Cloud AWS user, you have the ability to select different zones based on your region. Now, as per the latest webinar, Talend will expand the zones in the future. So keep this in mind. One of the other differences with Talend Cloud on AWS and Talend on Azure is that Talend's Data Preparation and Data Stewardship solution won't be available on Talend on Azure until the end of the year (2019). Luckily for you all, we only have another month until the end of Dec 2019, in which these solutions will be readily available for the Talend Cloud Azure users. Besides Talend Data Preparation and Data Stewardship not being available, Talend's API Designer and API Tester won't be available day 1. Though it was not discussed on the prior webinar, or I may have missed it, no date was given when Talend's API Designer and API Tester would be available on Talend Cloud on Azure. If anyone knows when this would be available, please let me know, else I will try to find out on the next live webinar. But though some of these applications may not be available, what is available is that Talend Cloud Azure users will be able to harness the cloud power of Microsoft Azure, in which Talend Cloud Azure runs natively. In addition, to access your TIC, you will have the ability to leverage Talend Studio and Talend's new self-service ETL solution, Pipeline Designer. This is a huge step for Talend and its Talend users and I ask you all, to check out and visit www.talend.com/talend-azure-trial to get access to your free 30 day trial to Talend Cloud on Azure. If you are looking for support and assistance with your Talend Cloud Azure 30 day trial, don't hesitate to reach out and if you are interested in learning more about Pingahla's Talend services, please contact sales@pingahla.com.