In SKYVVA, the bulk-loading option increases the performance of a session where huge volumes of data are involved. During bulk loading, the Integration Service bypasses the database log, which results in performance improvement. Without writing to the database log, however, the target database cannot perform a rollback. While using bulk loading, the need for improved session performance must be weighed against the ability to recover an incomplete session.
The Data Integration Service can use the SKYVVA Bulk API to write data to sObjects. Use the Bulk API to write large amounts of data to Salesforce with a minimal number of API calls. User can use the Bulk API to write data to Salesforce targets with Salesforce API.
With a Bulk API write, each batch of data can contain up to 10,000 records or one million characters of data in CSV format. When the Data Integration Service creates a batch, it adds required characters such as, quotation marks around text, to format the data.
User can configure a Bulk API target session to load batches serially or in parallel. By default, the data load is in parallel mode, but you can override the data load to serial mode. User can also monitor the progress of batches in the session log.
To configure a session to use the Bulk API for Salesforce targets, select the Use SKYVVA Bulk API session property. When user selects this property, the Data Integration Service ignores the Max Batch Size session property.
SKYVVA Bulk Interface Processing #
SKYVVA Bulk Interface Processing is an interface uses for running bulk attachments. When the user has much data, using Bulk Interface Processing is necessary (over 5000 records). Before configuring Bulk Interface Processing, the user has to know about the parameters:
BULK Mode: Check this flag if you want to run interface in a bulk mode using Salesforce BULK API.
Bulk Version: We support SFDC pure bulk mode in version 1.0 or 2.0. Then we have our own SKYVVA bulk mode. This uses SKYVVA workflow and mapping and thus is more powerful but slower than the pure bulk API technique from Salesforce. This one is SKYVVA’s own bulk mode.
Bulk Processing Mode: This field indicates the two-mode, which is support with bulk version 1.0. With bulk, version 2.0 only parallel mode is supported. Therefore, the value here depends on the selection of the field bulk version e.g. when we want to use ‘Serial’ then we needs to choose ‘Bulk V 1.0’. There are two modes which are:
- In figure we showing SKYVVA BULK Processing mode which include Parallel and series mode.
Parallel: When you select this processing mode, it will run parallel.
Serial: When you select this mode, it will not run at the same time.
Bulk Package Size: This parameter determines how many records are split into Bulk data size. If your Message contains 10.000 records and the value for this parameter is 1000 so the user will get 10 bulk data set in Salesforce.
Bulk Monitor Keep Size: The number of bulk executes logs to be kept.
Bulk Search Frequency: This is the schedule time frequency for the bulk scheduler on the interface.
Bulk Integrate Time Interval (minute): The interval (minutes) used to integrate bulk job scheduler. For example, if its value is 10, then the integrate bulk scheduler will be run every 10 minutes.
Bulk Mode #
In Bulk Mode, the Integration services bypass the database logs during job run for improved performance. There are two options of bulk mode, which is including SKYVVA bulk and SFDC bulk. The following section will be explained more detail about these two modes.
- SKYVVA Bulk
- SFDC Bulk
SKYVVA Bulk Mode #
In SKYVVA Bulk Mode user can create upsert, update, insert, and query, query All, pullQuery or delete a large volume of records with the Bulk API, which is optimized for processing large sets of data. It makes it simple to load, update, or delete data over a 5000 records.
SKYVVA Bulk is slow but very powerful, it allows you to do business mapping using our mapping tool and formula. Likewise, you can use the SKYVVA workflow to validate the data before posting. You can use SKYVVA Bulk mode with both Bulk-processing modes, which is Parallel and Serial. The default value is Serial.
Advantages #
- The major advantage of using SKYVVA bulk load is in the significant improvement of performance. Especially in the large volume table, bulk loading speeds up the process.
- If the SKYVVA job fails, the recovery is not possible due to the bypass of database logs. The only option during job failure is truncate and reprocess.
SFDC Bulk Mode #
We support SFDC pure bulk mode in version 2.0 that is very fast but cannot use the SKYVVA mapping tool to do business data mapping. On the other hand, SFDC Bulk requires the data has to be mapped already on the sender side and doing the additional mapping to enrich the business logic is not possible using SKYVVA mapping. This mode will work with both processing mode, Parallel and Serial, for version 1.0. Version 2.0, you can only use with Serial mode processing.
In SFDC Bulk Mode user can create update, or delete a large volume of records with the Bulk API, which is optimized for processing large sets of data. It makes it simple to load, update, or delete data from a few thousand to millions of records.
Difference Between SKYVVA Bulk and SFDC Bulk Mode? #
| SKYVVA Bulk Mode | SFDC Bulk Mode |
| In SKYVVA Bulk Mode user can create upsert, update, insert, and query, query All, pullQuery or delete a large volume of records with the Bulk API, which is optimized for processing large sets of data. It makes it simple to load, update, or delete data over a 5000 records. | In SFDC Bulk Mode user can create update, or delete a large volume of records with the Bulk API, which is optimized for processing large sets of data. It makes it simple to load, update, or delete data from a few thousand to millions of records. |
| Monitoring in Bulk Control Board: As user did integrate, your data you might want to check or monitor if all your data are successfully integrated. To monitor bulk data. There are three orders to monitor bulk processing: Bulk Data Inbox : When data pushed from the client using Bulk API, that will appear in this section Bulk Data Processing: Bulk Data Processing is using to store data that are processing. Bulk Monitor: User can monitor the total records of attachments, total batch and batches that had processed. |
Monitoring Bulk Data Load Jobs: Process a set of records by creating a job that contains data that will be processed asynchronously. The job specifies which object is being processed and what type of operation is being used. |
| Indexes, constraints, auto-generated features can be available in table level during the job run.
Operations:
It support SFDC BulkV1.0,SFDCV2.0 for all Job Type |
Operations:
hardDelete (Bulk V1 type jobs only) |
Use case to Use Bulk Mode #
Using SKYVVA Bulk version
- Create Integration
- Create Interface
- Click on Interface
- Scroll down to Bulk Setting section
- Check flag of BULK Mode
- Select SKYVVA Bulk1.0 valuefrom picklist
- Save
- Scroll down to Source Definition section
- Choose file & Save
- Click on Open Mapping Button
- Select Ext Id
- Save
Agent Control Board:
Go to your Salesforce Org.
- Click on setup
- Write Remote Site on the Search box
- Click on New Remote Site button
- Great user-created New Remote site successfully
Go to Agent Control Board Tab
- Click on your AGENT Properties File under Configuration Tab
- See Status is OFFLINE
- Click on the continue button
- Users can see now AGENT properties file import the Salesforce Credential successfully
- Username, Security token, Password and also status ONLINE
- Click on Save button now
Click on ping connection.
Create adapter.
- Go to adapter tab -> click on New button.
- Create File adapter with appropriate details and save it.
Add this adapter in interface.
Go to Interface tab to process the interface.
Go to Bulk control board and check the processing basket.
