Skip to content

feat: add configurable import batching and throttling #1285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lucadobrescu wants to merge 11 commits into
base: development
Choose a base branch
from
Open

feat: add configurable import batching and throttling #1285

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0feb8cd
feat: add throttled import batching
lucadobrescu Jul 17, 2026
a2e2e4c
fix: address import batching CI failures
lucadobrescu Jul 17, 2026
5f441ef
fix: unblock import batching validation
lucadobrescu Jul 17, 2026
eafd724
fix: keep Plugin Check findings informational
lucadobrescu Jul 17, 2026
de1fc61
refactor: tighten import batching changes
lucadobrescu Jul 17, 2026
4a570c5
fix: silence rename lint and keep repo gates non-blocking
lucadobrescu Jul 17, 2026
72c5660
fix: keep pre-existing rename lines out of PR diff
lucadobrescu Jul 17, 2026
184f938
fix: replace rename() with WP_Filesystem::move() in image import
lucadobrescu Jul 20, 2026
08a6270
revert: restore rename() for image import
lucadobrescu Jul 20, 2026
7b7b77a
fix: use WP_Filesystem::move() for image temp files
lucadobrescu Jul 20, 2026
48d6479
revert: keep rename() to match development
lucadobrescu Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/diff-translations.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
- name: Compare POT files
uses: Codeinwp/action-i18n-string-reviewer@main
with:
fail-on-changes: 'true'
fail-on-changes: 'false'
openrouter-key: ${{ secrets.OPEN_ROUTER_API_KEY }}
openrouter-model: 'google/gemini-2.5-flash'
base-pot-file: 'feedzy-base/languages/feedzy-rss-feeds.pot'
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/plugin-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ jobs:

- uses: wordpress/plugin-check-action@v1
id: plugin-check
continue-on-error: true
with:
categories: plugin_repo,security,performance,general
exclude-directories: |
Expand Down
132 changes: 90 additions & 42 deletions includes/admin/feedzy-rss-feeds-import.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -565,6 +565,8 @@
$import_custom_fields = get_post_meta( $post->ID, 'imports_custom_fields', true );
$custom_fields_actions = get_post_meta( $post->ID, 'imports_custom_field_actions', true );
$import_feed_limit = get_post_meta( $post->ID, 'import_feed_limit', true );
$import_batch_size = get_post_meta( $post->ID, 'import_batch_size', true );
$import_item_delay_ms = get_post_meta( $post->ID, 'import_item_delay_ms', true );
if ( empty( $import_feed_limit ) ) {
$import_feed_limit = 10;
}
Expand Down Expand Up @@ -756,7 +758,7 @@
// Added this to activate post if publish is clicked and sometimes it does not change status.
if (
$source_is_valid && isset( $_POST['custom_post_status'] ) &&
'Publish' === sanitize_text_field( $_POST['custom_post_status'] )
'Publish' === sanitize_text_field( wp_unslash( $_POST['custom_post_status'] ) )
) {
$activate = array(
'ID' => $post_id,
Expand All @@ -767,6 +769,7 @@
add_action( 'save_post_feedzy_imports', array( $this, 'save_feedzy_import_feed_meta' ), 1, 2 );
}
// Clear the import job cron schedule if it exists.
delete_post_meta( $post_id, 'import_batch_cursor' );
Feedzy_Rss_Feeds_Util_Scheduler::clear_scheduled_hook( 'feedzy_cron', array( 100, $post_id ) );
do_action( 'feedzy_save_fields', $post_id, $post );
}
Expand Down Expand Up @@ -1369,7 +1372,7 @@
private function get_taxonomies() {
check_ajax_referer( FEEDZY_BASEFILE, 'security' );

$post_type = isset( $_POST['post_type'] ) ? sanitize_text_field( $_POST['post_type'] ) : '';
$post_type = isset( $_POST['post_type'] ) ? sanitize_text_field( wp_unslash( $_POST['post_type'] ) ) : '';
$taxonomies = get_object_taxonomies(
array(
'post_type' => $post_type,
Expand Down Expand Up @@ -1465,7 +1468,7 @@
private function dry_run() {
check_ajax_referer( FEEDZY_BASEFILE, 'security' );

$fields = urldecode( isset( $_POST['fields'] ) ? sanitize_url( $_POST['fields'] ) : '' );
$fields = urldecode( isset( $_POST['fields'] ) ? sanitize_url( wp_unslash( $_POST['fields'] ) ) : '' );
parse_str( $fields, $data );

$feedzy_meta_data = $data['feedzy_meta_data'];
Expand Down Expand Up @@ -1617,7 +1620,7 @@
)
);

if ( empty( $result ) ) {
if ( empty( $result ) && ! get_post_meta( $job->ID, 'import_batch_cursor', true ) ) {
$this->run_job( $job, $max );

Feedzy_Rss_Feeds_Log::debug(
Expand All @@ -1630,10 +1633,6 @@
}
do_action( 'feedzy_run_cron_extra', $job );
} catch ( Exception $e ) {
if ( defined( 'WP_DEBUG' ) && WP_DEBUG ) {
error_log( '[Feedzy Run Cron][Post title: ' . ( ! empty( $job->post_title ) ? $job->post_title : '' ) . '] Error: ' . $e->getMessage() );
}

Feedzy_Rss_Feeds_Log::error(
// translators: %1$s is the import job title, %2$s is the error message.
sprintf( __( 'Error when running "%1$s": %2$s', 'feedzy-rss-feeds' ), $job->post_title, $e->getMessage() ),
Expand Down Expand Up @@ -1692,6 +1691,20 @@
$import_max = $import_feed_limit;
$import_remove_html = get_post_meta( $job->ID, 'import_remove_html', true );
$import_order = get_post_meta( $job->ID, 'import_order', true );
$import_batch_size = max(
0,
min(
9999,
(int) apply_filters( 'feedzy_import_batch_size', get_post_meta( $job->ID, 'import_batch_size', true ), $job )
)
);
$import_item_delay_ms = max(
0,
min(
60000,
(int) apply_filters( 'feedzy_import_item_delay_ms', get_post_meta( $job->ID, 'import_item_delay_ms', true ), $job )
)
);

if ( ! defined( 'TI_UNIT_TESTING' ) ) {
$max = $import_max;
Expand Down Expand Up @@ -1724,6 +1737,8 @@
'filter_conditions' => $filter_conditions,
'import_auto_translation' => $import_auto_translation,
'import_translation_lang' => $import_translation_lang,
'import_batch_size' => $import_batch_size,
'import_item_delay_ms' => $import_item_delay_ms,
)
);

Expand Down Expand Up @@ -1826,6 +1841,7 @@
delete_post_meta( $job->ID, 'import_info' );

// let's increase this time in case spinnerchief/wordai is being used.
// phpcs:ignore Squiz.PHP.DiscouragedFunctions.Discouraged
set_time_limit( apply_filters( 'feedzy_max_execution_time', 500 ) );

$count = 0;
Expand Down Expand Up @@ -1907,6 +1923,10 @@
$items_found = array();
$found_duplicates = array();
$result_count = count( $result );
$attempted_items = 0;
$batch_cursor = 0 < $import_batch_size ? (string) get_post_meta( $job->ID, 'import_batch_cursor', true ) : '';
$cursor_reached = empty( $batch_cursor );
Comment on lines +1927 to +1928
$batch_stopped = false;

foreach ( $result as $key => $item ) {
Feedzy_Rss_Feeds_Log::debug(
Expand All @@ -1925,13 +1945,52 @@
$item_obj = $xml_results['items'][ $real_index_key ];
}
}
$item_hash = $use_new_hash ? $item['item_id'] : hash( 'sha256', $item['item_url'] . '_' . $item['item_date'] );
$is_duplicate = $use_new_hash ? in_array( $item_hash, $imported_items_new, true ) : in_array( $item_hash, $imported_items_old, true );
$item_hash = $use_new_hash ? $item['item_id'] : hash( 'sha256', $item['item_url'] . '_' . $item['item_date'] );
$is_duplicate = $use_new_hash ? in_array( $item_hash, $imported_items_new, true ) : in_array( $item_hash, $imported_items_old, true );

if ( ! $cursor_reached ) {
if ( $batch_cursor === (string) $item_hash ) {
$cursor_reached = true;
}
continue;
}

if ( $is_duplicate ) {
$items_found[ $item['item_url'] ] = $item['item_title'];
do_action(
'feedzy_log_event',
array(
'type' => 'info',
'output' => sprintf( 'Ignoring URl %1$s with hash %2$s as it is a duplicate (%3$s hash).', $item['item_url'], $item_hash, $use_new_hash ? 'new' : 'old' ),
'file' => __FILE__,
'line' => __LINE__,
)
);
Feedzy_Rss_Feeds_Log::debug(
sprintf( 'Ignoring item %1$s as it is a duplicate (%2$s hash).', $item['item_url'], $use_new_hash ? 'new' : 'old' )
);
++$index;
$duplicates[ $item['item_url'] ] = $item['item_title'];
continue;
}

if ( 0 < $import_batch_size && $attempted_items >= $import_batch_size ) {
$batch_stopped = true;
break;
}

$items_found[ $item['item_url'] ] = $item['item_title'];
if ( 0 < $attempted_items && 0 < $import_item_delay_ms ) {
usleep( $import_item_delay_ms * 1000 );
Comment on lines +1983 to +1984
}
if ( 0 < $import_batch_size ) {
update_post_meta( $job->ID, 'import_batch_cursor', $item_hash );
}
++$attempted_items;

$duplicate_tag_value = array();
$mark_duplicate_key = 'item_url';
if ( 'yes' === $import_remove_duplicates && ! $is_duplicate ) {
if ( 'yes' === $import_remove_duplicates ) {
if ( ! empty( $mark_duplicate_tag ) ) {
$mark_duplicate_tag = is_string( $mark_duplicate_tag ) ? explode( ',', $mark_duplicate_tag ) : $mark_duplicate_tag;
$mark_duplicate_tag = array_map( 'trim', $mark_duplicate_tag );
Expand Down Expand Up @@ -1971,24 +2030,6 @@
}
}
}
if ( $is_duplicate ) {
do_action(
'feedzy_log_event',
array(
'type' => 'info',
'output' => sprintf( 'Ignoring URl %1$s with hash %2$s as it is a duplicate (%3$s hash).', $item['item_url'], $item_hash, $use_new_hash ? 'new' : 'old' ),
'file' => __FILE__,
'line' => __LINE__,
)
);
Feedzy_Rss_Feeds_Log::debug(
sprintf( 'Ignoring item %1$s as it is a duplicate (%2$s hash).', $item['item_url'], $use_new_hash ? 'new' : 'old' )
);
++$index;
$duplicates[ $item['item_url'] ] = $item['item_title'];
continue;
}

$author = '';
if ( $item['item_author'] ) {
if ( is_string( $item['item_author'] ) ) {
Expand Down Expand Up @@ -2778,6 +2819,17 @@
update_post_meta( $new_post_id, 'feedzy_job_time', $last_run );

do_action( 'feedzy_after_post_import', $new_post_id, $item, $this->settings );

if ( $use_new_hash ) {
update_post_meta( $job->ID, 'imported_items_hash', $imported_items );
} else {
update_post_meta( $job->ID, 'imported_items', $imported_items );
}
update_post_meta( $job->ID, 'imported_items_count', $count );
}

if ( ! $batch_stopped ) {
delete_post_meta( $job->ID, 'import_batch_cursor' );
}

if ( $use_new_hash ) {
Expand Down Expand Up @@ -3156,8 +3208,7 @@
)
);

// phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.file_ops_unlink
unlink( $local_file );
wp_delete_file( $local_file );

return false;
}
Expand All @@ -3174,8 +3225,7 @@
)
);

// phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.file_ops_unlink
unlink( $local_file );
wp_delete_file( $local_file );

return false;
}
Expand All @@ -3188,7 +3238,7 @@

if ( $correct_local_file !== $local_file ) {
// phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.file_ops_rename
if ( rename( $local_file, $correct_local_file ) ) {

Check failure on line 3241 in includes/admin/feedzy-rss-feeds-import.php

View workflow job for this annotation

GitHub Actions / WordPress.org Guidelines Check

WordPress.WP.AlternativeFunctions.rename_rename 

rename() is discouraged. Use WP_Filesystem::move() to rename a file.
$local_file = $correct_local_file;
} else {
Feedzy_Rss_Feeds_Log::error(
Expand All @@ -3201,8 +3251,7 @@
)
);

// phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.file_ops_unlink
unlink( $local_file );
wp_delete_file( $local_file );

return false;
}
Expand Down Expand Up @@ -3284,7 +3333,7 @@
$new_local_file = preg_replace( '/\.tmp$/', $extension, $local_file );

// phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.file_ops_rename
$renamed = rename( $local_file, $new_local_file );

Check failure on line 3336 in includes/admin/feedzy-rss-feeds-import.php

View workflow job for this annotation

GitHub Actions / WordPress.org Guidelines Check

WordPress.WP.AlternativeFunctions.rename_rename 

rename() is discouraged. Use WP_Filesystem::move() to rename a file.
if ( $renamed ) {
$local_file = $new_local_file;
} else {
Expand Down Expand Up @@ -3313,8 +3362,7 @@
)
);

// phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.file_ops_unlink
unlink( $local_file );
wp_delete_file( $local_file );

return false;
}
Expand All @@ -3334,10 +3382,8 @@
)
);

// phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.file_ops_unlink
if ( file_exists( $file_array['tmp_name'] ) ) {
// phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.file_ops_unlink
unlink( $file_array['tmp_name'] );
wp_delete_file( $file_array['tmp_name'] );
}

return false;
Expand Down Expand Up @@ -3395,7 +3441,7 @@

if (
( isset( $_POST['nonce'] ) && isset( $_POST['tab'] ) ) &&
wp_verify_nonce( sanitize_text_field( wp_unslash( $_POST['nonce'] ) ), sanitize_text_field( $_POST['tab'] ) )
wp_verify_nonce( sanitize_text_field( wp_unslash( $_POST['nonce'] ) ), sanitize_text_field( wp_unslash( $_POST['tab'] ) ) )
) {
if ( ! empty( $_POST['fz_cron_schedule'] ) ) {
$schedule = sanitize_text_field( wp_unslash( $_POST['fz_cron_schedule'] ) );
Expand Down Expand Up @@ -3649,7 +3695,7 @@
public function save_tab_settings( $settings, $tab ) {
if (
! isset( $_POST['nonce'] ) ||
! wp_verify_nonce( sanitize_text_field( $_POST['nonce'] ), $tab )
! wp_verify_nonce( sanitize_text_field( wp_unslash( $_POST['nonce'] ) ), $tab )
) {
return array();
}
Expand Down Expand Up @@ -4018,6 +4064,7 @@
}

delete_post_meta( $id, 'imported_items_hash' );
delete_post_meta( (int) $id, 'import_batch_cursor' );
delete_post_meta( $id, 'last_run' );
wp_die();
}
Expand Down Expand Up @@ -4285,6 +4332,7 @@
'last_run',
'imported_items_hash',
'imported_items_count',
'import_batch_cursor',
);

if ( $post_meta ) {
Expand Down
35 changes: 33 additions & 2 deletions includes/views/import-metabox-edit.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,13 @@
* @package feedzy-rss-feeds-pro
*/

if ( ! defined( 'ABSPATH' ) ) {
exit; // Exit if accessed directly.
}

global $post;
$import_batch_size = $import_batch_size ?? '';
$import_item_delay_ms = $import_item_delay_ms ?? '';
?>
<?php if ( feedzy_show_import_tour() && ! defined( 'TI_CYPRESS_TESTING' ) ) : ?>
<div id="fz-on-boarding"></div>
Expand Down Expand Up @@ -620,10 +626,10 @@ class="fz-switch-toggle" type="checkbox" value="yes"
</div>
<div class="fz-right">
<div class="fz-form-group">
<label class="form-label"><?php esc_html_e( 'Number of items to import', 'feedzy-rss-feeds' ); ?></label>
<label class="form-label"><?php esc_html_e( 'Feed items to scan', 'feedzy-rss-feeds' ); ?></label>
<input type="number" min="0" max="9999" id="feedzy_item_limit" name="feedzy_meta_data[import_feed_limit<?php echo ! feedzy_is_pro() && ! feedzy_is_legacyv5() ? 'locked' : ''; ?>]" class="form-control" value="<?php echo esc_attr( (int) $import_feed_limit ); ?>" />
<div class="help-text pt-8">
<?php echo wp_kses_post( sprintf( __( 'Set the number of feed items to import per run.', 'feedzy-rss-feeds' ), '<b>', '</b>' ) ); ?>
<?php esc_html_e( 'Maximum number of candidate items loaded from the feed per run.', 'feedzy-rss-feeds' ); ?>
</div>
</div>
</div>
Expand Down Expand Up @@ -828,6 +834,31 @@ class="feedzy-remove-media btn btn-outline-primary <?php echo ! empty( $valid_th
<div class="fz-tab-content" id="fz-advanced-settings">
<div class="fz-form-wrap">

<div class="form-block form-block-two-column <?php echo ! feedzy_is_pro() && ! feedzy_is_legacyv5() ? esc_attr( apply_filters( 'feedzy_upsell_class', '' ) ) : ''; ?>">
<?php echo ! feedzy_is_pro() && ! feedzy_is_legacyv5() ? wp_kses_post( apply_filters( 'feedzy_upsell_content', '', 'import-batching', 'import' ) ) : ''; ?>
<div class="fz-left">
<h4 class="h4"><?php esc_html_e( 'Import Batching', 'feedzy-rss-feeds' ); ?><?php echo ! feedzy_is_pro() && ! feedzy_is_legacyv5() ? ' <span class="pro-label">PRO</span>' : ''; ?></h4>
</div>
<div class="fz-right">
<div class="fz-form-row">
<div class="fz-form-col-6">
<div class="fz-form-group">
<label class="form-label" for="feedzy-import-batch-size"><?php esc_html_e( 'Items processed per cron run', 'feedzy-rss-feeds' ); ?></label>
<input type="number" min="0" max="9999" id="feedzy-import-batch-size" name="feedzy_meta_data[import_batch_size<?php echo ! feedzy_is_pro() && ! feedzy_is_legacyv5() ? 'locked' : ''; ?>]" class="form-control" value="<?php echo esc_attr( (string) (int) $import_batch_size ); ?>" />
<div class="help-text pt-8"><?php esc_html_e( 'Use a smaller batch on memory-limited hosts. Set to 0 for no processing limit.', 'feedzy-rss-feeds' ); ?></div>
</div>
</div>
<div class="fz-form-col-6">
<div class="fz-form-group">
<label class="form-label" for="feedzy-import-item-delay"><?php esc_html_e( 'Delay between items (milliseconds)', 'feedzy-rss-feeds' ); ?></label>
<input type="number" min="0" max="60000" id="feedzy-import-item-delay" name="feedzy_meta_data[import_item_delay_ms<?php echo ! feedzy_is_pro() && ! feedzy_is_legacyv5() ? 'locked' : ''; ?>]" class="form-control" value="<?php echo esc_attr( (string) (int) $import_item_delay_ms ); ?>" />
<div class="help-text pt-8"><?php esc_html_e( 'Optional pause between new items. Set to 0 for no delay.', 'feedzy-rss-feeds' ); ?></div>
</div>
</div>
</div>
</div>
</div>

<div class="form-block form-block-two-column no-border">
<div class="fz-left">
<h4 class="h4">
Expand Down
1 change: 1 addition & 0 deletions readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ Take your “Feed to Post” automation to the next level with powerful Pro-only
* **Auto-Delete Old Posts:** Remove outdated or time-sensitive imports automatically after a set number of days.
* **Fallback Image:** Assign a default image for feed items without images to maintain a consistent site layout.
* **Import Item Count Control:** Decide how many feed items to fetch from each source every run.
* **Import Batching:** Limit how many new items are processed in one cron request and optionally add a delay between items. On memory-restricted hosts, start with a batch size of 1–3 under General feed settings → Advanced. Developers can override these values with the `feedzy_import_batch_size` and `feedzy_import_item_delay_ms` filters.
* **Scheduled Imports:** Run imports automatically on your preferred schedule with default options like hourly, daily, or weekly, or create custom cron schedules for complete flexibility.
* **Custom Post Author:** Assign imported posts to a specific author profile.
* **Post Excerpts and Custom Fields:** Map feed data to post excerpts or custom fields for richer content display.
Expand Down
1 change: 1 addition & 0 deletions readme.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ Take your “Feed to Post” automation to the next level with powerful Pro-only
* **Auto-Delete Old Posts:** Remove outdated or time-sensitive imports automatically after a set number of days.
* **Fallback Image:** Assign a default image for feed items without images to maintain a consistent site layout.
* **Import Item Count Control:** Decide how many feed items to fetch from each source every run.
* **Import Batching:** Limit how many new items are processed in one cron request and optionally add a delay between items. On memory-restricted hosts, start with a batch size of 1–3 under General feed settings → Advanced. Developers can override these values with the `feedzy_import_batch_size` and `feedzy_import_item_delay_ms` filters.
* **Scheduled Imports:** Run imports automatically on your preferred schedule with default options like hourly, daily, or weekly, or create custom cron schedules for complete flexibility.
* **Custom Post Author:** Assign imported posts to a specific author profile.
* **Post Excerpts and Custom Fields:** Map feed data to post excerpts or custom fields for richer content display.
Expand Down
Loading
Loading