Apache Nifi

Empowering Data-Driven Organizations with Cloudera Flow Management 4 (powered by Apache NiFi 2.0) Apache NiFi has long been a cornerstone for data engineering, providing a powerful and flexible framework for data ingestion, transformation, and distribution. As a leading contributor to NiFi, Cloudera has been instrumental in driving its evolution and adoption. With the recent release of Cloudera Flow Management 4.0 in Technical Preview as the first NiFi 2.0-based Cloudera Flow Management release, we are excited to showcase the enhanced capabilities and how Cloudera continues to lead the way in data flow management.The Value of NiFi 2.0 and Cloudera Flow Management 4.0Cloudera Flow Management 4.0 (powered by Apache NiFi 2.0) introduces significant improvements, including:Enhanced Performance: NiFi 2.0 boasts significant performance enhancements, handling data flows more efficiently and scaling to larger workloads. These enhancements give users more power and reliability to ingest, process, and distribute larger and more complex data sets.Streamlined Development: The new flow canvas interface and improved drag-and-drop functionality make flow development faster and more intuitive. This significantly decreases flow development time, leading to cost savings.Advanced Security: NiFi 2.0 introduces enhanced security features, including improved encryption and authentication mechanisms. This provides more confidence in a secure and reliable system for processing sensitive data.Expanded Integrations: NiFi 2.0 seamlessly integrates with a wider range of data sources and systems, expanding its applicability across various use cases. Cloudera Flow Management 4.0 specifically retains components to support integrations to applications in Cloudera where many components such as Hive and Accumulo were removed in Apache NiFi 2.0. In addition, Cloudera Flow Management 4.0 includes new integrations such as Change Data Capture (CDC) capabilities for relational database systems as well as Iceberg. This allows users to design their own end-to-end systems using Cloudera applications as well as external systems .Native Python Processor Development: NiFi 2.0 provides a Python SDK for which processors can be rapidly developed in Python and deployed in flows. Some common document parsing processors written in Python are included in the release. Cloudera Flow Management 4.0 specifically adds components for embedding data, ingesting into vector databases, prompting several GenAI systems and working with Large Language Models (LLMs) via Amazon Bedrock. This provides users with an impressive set of GenAI capabilities to empower their business cases.Best Practices in Flow Design: NiFi 2.0 provides a rules engine for developing flow analysis rules that recommend and enforce best practices for flow design. Cloudera Flow Management 4.0

Provides several Flow Analysis Rules for such aspects as thread management and recommended components. Cloudera Flow Management administrators can leverage these to ensure well-designed and robust flows for their use cases.Cloudera and NiFi - Continued Support, Innovation, and Simplified Migration Cloudera has been a driving force behind NiFi's development, actively contributing to its open-source community and providing expert guidance to users. Cloudera has invested heavily in NiFi, ensuring its continued evolution and relevance in the ever-changing data landscape.Our commitment to NiFi is evident in our initiatives. We actively participate in the Apache NiFi community, sharing knowledge, best practices and supporting users through mailing lists, forums, and events. In addition to community contributions, Cloudera Flow Management Operator enables customers to deploy and manage NiFi clusters and NiFi Registry instances on Kubernetes application platforms. Cloudera Flow Management Operator simplifies data collection, transformation, and delivery across enterprises. Leveraging containerized infrastructure, the operator streamlines the orchestration of complex data flows. Cloudera is the only provider with a Migration Tool that simplifies the complex and repetitive process of migrating Cloudera Flow Management flows from the NiFi 1 set of components to use the NiFi 2 set. To these ends, Cloudera provides comprehensive training and consulting services to help organizations leverage the full potential of NiFi.Driving the Future of Data Flow ManagementWith Cloudera Flow Management 4.0.0 (powered by Apache NiFi 2.0), Cloudera fortifies its leadership in data flow management. We will continue to invest in NiFi's development, ensuring it remains a powerful and reliable tool for data engineers and data scientists. In addition, Cloudera provides cloud-based deployments of Cloudera Flow Management, optimizing your operational efficiency and allowing you to scale to the enterprise with confidence. Features enabling, integrating with, and enhancing your AI-based solutions are a central focus of Cloudera Flow Management. We also continue to provide support and guidance to our customers, helping them harness the full power of NiFi to drive business-critical data initiatives.Learn More:To explore the new capabilities of Cloudera Flow Management and discover how it can transform your data pipelines, learn more here:Data Distribution Architecture to Drive InnovationScaling NiFi for the Enterprise with Cloudera

Apache NiFi is an open-source workflow management software designed to automate data flow between software systems. It allows the creation of ETL data pipelines and is shipped with more than 300 data processors. This step-by-step tutorial shows how to connect Apache NiFi to ClickHouse as both a source and destination, and to load a sample dataset.1. Gather your connection details​To connect to ClickHouse with HTTP(S) you need this information:The HOST and PORT: typically, the port is 8443 when using TLS or 8123 when not using TLS.The DATABASE NAME: out of the box, there is a database named default, use the name of the database that you want to connect to.The USERNAME and PASSWORD: out of the box, the username is default. Use the username appropriate for your use case.The details for your ClickHouse Cloud service are available in the ClickHouse Cloud console. Select the service that you will connect to and click Connect:Choose HTTPS, and the details are available in an example curl command.If you are using self-managed ClickHouse, the connection details are set by your ClickHouse administrator.2. Download and run Apache NiFi​For a new setup, download the binary from and start by running ./bin/nifi.sh start3. Download the ClickHouse JDBC driver​Visit the ClickHouse JDBC driver release page on GitHub and look for the latest JDBC release versionIn the release version, click on "Show all xx assets" and look for the JAR file containing the keyword "shaded" or "all", for example, clickhouse-jdbc-0.5.0-all.jarPlace the JAR file in a folder accessible by Apache

NiFi and take note of the absolute path4. Add DBCPConnectionPool Controller Service and configure its properties​To configure a Controller Service in Apache NiFi, visit the NiFi Flow Configuration page by clicking on the "gear" buttonSelect the Controller Services tab and add a new Controller Service by clicking on the + button at the top rightSearch for DBCPConnectionPool and click on the "Add" buttonThe newly added DBCPConnectionPool will be in an Invalid state by default. Click on the "gear" button to start configuringUnder the "Properties" section, input the following valuesPropertyValueRemarkDatabase Connection URLjdbc:ch: HOSTNAME in the connection URL accordinglyDatabase Driver Class Namecom.clickhouse.jdbc.ClickHouseDriverDatabase Driver Location(s)/etc/nifi/nifi-X.XX.X/lib/clickhouse-jdbc-0.X.X-patchXX-shaded.jarAbsolute path to the ClickHouse JDBC driver JAR fileDatabase UserdefaultClickHouse usernamePasswordpasswordClickHouse passwordIn the Settings section, change the name of the Controller Service to "ClickHouse JDBC" for easy referenceActivate the DBCPConnectionPool Controller Service by clicking on the "lightning" button and then the "Enable" buttonCheck the Controller Services tab and ensure that the Controller Service is enabled5. Read from a table using the ExecuteSQL processor​Add an ​​ExecuteSQL processor, along with the appropriate upstream and downstream processorsUnder the "Properties" section of the ​​ExecuteSQL processor, input the following valuesPropertyValueRemarkDatabase Connection Pooling ServiceClickHouse JDBCSelect the Controller Service configured for ClickHouseSQL select querySELECT * FROM system.metricsInput your query hereStart the ​​ExecuteSQL processorTo confirm that the query has been processed successfully, inspect one of the FlowFile in the output queueSwitch view to "formatted" to view the result of the output FlowFile6. Write to a table using MergeRecord and PutDatabaseRecord processor​To write multiple rows in a single

Insert, we first need to merge multiple records into a single record. This can be done using the MergeRecord processorUnder the "Properties" section of the MergeRecord processor, input the following valuesPropertyValueRemarkRecord ReaderJSONTreeReaderSelect the appropriate record readerRecord WriterJSONReadSetWriterSelect the appropriate record writerMinimum Number of Records1000Change this to a higher number so that the minimum number of rows are merged to form a single record. Default to 1 rowMaximum Number of Records10000Change this to a higher number than "Minimum Number of Records". Default to 1,000 rowsTo confirm that multiple records are merged into one, examine the input and output of the MergeRecord processor. Note that the output is an array of multiple input recordsInputOutputUnder the "Properties" section of the PutDatabaseRecord processor, input the following valuesPropertyValueRemarkRecord ReaderJSONTreeReaderSelect the appropriate record readerDatabase TypeGenericLeave as defaultStatement TypeINSERTDatabase Connection Pooling ServiceClickHouse JDBCSelect the ClickHouse controller serviceTable NametblInput your table name hereTranslate Field NamesfalseSet to "false" so that field names inserted must match the column nameMaximum Batch Size1000Maximum number of rows per insert. This value should not be lower than the value of "Minimum Number of Records" in MergeRecord processorTo confirm that each insert contains multiple rows, check that the row count in the table is incrementing by at least the value of "Minimum Number of Records" defined in MergeRecord.Congratulations - you have successfully loaded your data into ClickHouse using Apache NiFi !

To the cloud.● Key features: Application anddata dependency mapping, automated migration workflows, testing and validationcapabilities.● Examples: Azure Migrate, GoogleCloud Migrate for Compute Engine. 4. Cloud migration tools● Purpose: Facilitate the transferof data and applications from on-premises environments to cloud platforms orbetween cloud environments.● Key features: Secure datatransfer, support for various cloud platforms, scalability and minimaldowntime.● Examples: AWS Migration Hub,Azure Site Recovery, Google Cloud Transfer Service, Acronis Cyber Protect. 5. Data integration tools● Purpose: Combine data fromdifferent sources into a single, unified view, often used in data warehousingand ETL (extract, transform, load) processes.● Key features: Data extraction,transformation and loading capabilities; support for various data sources andformats; data quality and cleansing functions.● Examples: Talend, InformaticaPowerCenter, Apache NiFi. 6. Big data migration tools● Purpose: Handle the migration oflarge volumes of data, often involving complex data structures and high-speedtransfer requirements.● Key features: Scalability,parallel processing, support for big data platforms like Hadoop and Spark,robust error handling.● Examples: Apache Sqoop, IBM BigReplicate. 7. Content management system(CMS) migration tools● Purpose: Migrate content fromone CMS to another, commonly used in website redesigns or platform upgrades.● Key features: Content mapping,media transfer, link redirection, metadata preservation.● Examples: WordPress WP AllImport, CMS2CMS. 8. Email migration tools● Purpose: Transfer email datafrom one email system to another, such as migrating from on-premises Exchangeto Office 365.● Key features: Mailbox transfer,calendar and contact migration, email formatting preservation, secure datahandling.● Examples: Microsoft ExchangeMigration, Google Workspace Migrate, AcronisCyber Protect. 9. Data replication tools● Purpose: Continuously replicatedata from one system to another, often used for real-time data synchronizationand disaster recovery.● Key features: Real-time datareplication, conflict resolution, minimal latency, support for various datasources.● Examples: HVR, Qlik Replicate,IBM InfoSphere Data Replication, AcronisCyber Protect. 10. Hybrid migration tools● Purpose: Handle multiple typesof migration scenarios within a single platform, offering versatility andcomprehensive capabilities.● Key features: Multi-sourcesupport, integrated data transformation, user-friendly interfaces, robust