I currently have a puppet 3.7.2 installation that’s still using storeconfigs (yes, I know it’s a deprecated feature). Exported resources contained in the storeconfigs database have the anonying property of never expiring. Under normal operation they will persist up until the point that the node is decomissioned via the node face. Eg.: puppet node clean hostname.example.org
That mechanism falls down in the scenario that the node is not being decomissioned and the manifest has changed to no longer contain the exported resource(s). Running puppet node clean ... purges not just exported resources but also invalidates the agent’s x509 certificate. This is a tad overhanded and unfortunately the node face does not have an option to invoke only the #clean_storeconfigs method. I decided it wasn’t worth the effort to add one as it would be unlikely to get merged upstream since the entire facility is about to be removed.
I briefly attempted to get my head around what ActiveRecord was doing to the database when a node was purged by code inspection but decided it would be more expedient to cheat and observe the queries directly.
Mysql has the ability to enable/disable a query log dynamically. This is global and grabs absolutely everything reguardless of the database namespace. It will be both difficult to fish out the queries your interested and grow rapidly on a busy instance. In my case, I had to briefly stop the puppet master in order to get the scroll shock down to a reasonable volume.
SET GLOBAL log_output = "FILE"; SET GLOBAL general_log_file = "/tmp/my.log"; SET GLOBAL general_log = 'ON';
I then purged a node that had already been shutdown.
# puppet node clean hostname.example.org Notice: Revoked certificate with serial 289 Notice: hostname.example.org storeconfigs removed hostname.example.org
and quickly disabled the mysql global log.
SET GLOBAL general_log = 'OFF';
This is the what the query trace looked like:
1063 Connect puppet@localhost on puppet
1063 Query SET SQL_AUTO_IS_NULL=0
1063 Statistics
1063 Query SHOW FIELDS FROM `hosts`
1063 Query SHOW TABLES
1063 Query SHOW FIELDS FROM `hosts`
1063 Query SELECT `hosts`.* FROM `hosts` WHERE `hosts`.`name` = 'hostname.example.org' LIMIT 1
1063 Query BEGIN
1063 Query SELECT `fact_values`.* FROM `fact_values` WHERE (`fact_values`.host_id = 45)
1063 Query SHOW FIELDS FROM `fact_values`
1063 Query SHOW FIELDS FROM `fact_values`
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558422
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558423
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558424
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558425
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558426
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558427
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558428
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558429
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558430
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558431
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558432
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558433
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558434
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558435
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558436
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558437
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558438
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558439
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558440
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558441
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558442
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558443
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558444
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558445
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558446
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558447
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558448
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558449
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558450
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558451
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558452
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558453
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558454
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558455
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558456
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558457
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558458
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558459
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558460
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558461
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558462
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558463
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558464
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558465
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558466
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558467
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558468
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558469
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558470
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558471
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558472
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558473
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558474
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558475
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558476
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558477
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558478
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558479
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558480
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558481
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558482
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558483
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558484
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558485
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558486
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558487
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558488
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558489
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558490
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558491
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558492
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558493
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558494
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558495
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558496
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558497
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558498
1063 Query DELETE FROM `fact_values` WHERE `fact_values`.`id` = 2558499
1063 Query SELECT `resources`.* FROM `resources` WHERE (`resources`.host_id = 45)
1063 Query SHOW FIELDS FROM `resources`
1063 Query SELECT `param_values`.* FROM `param_values` WHERE (`param_values`.resource_id = 117)
1063 Query SHOW FIELDS FROM `param_values`
1063 Query SHOW FIELDS FROM `param_values`
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 694
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 695
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 696
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 697
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 698
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 699
1063 Query SELECT `resource_tags`.* FROM `resource_tags` WHERE (`resource_tags`.resource_id = 117)
1063 Query SHOW FIELDS FROM `resource_tags`
1063 Query SHOW FIELDS FROM `resource_tags`
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1249
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1250
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1251
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1252
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1253
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1254
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1255
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1256
1063 Query SHOW FIELDS FROM `resources`
1063 Query DELETE FROM `resources` WHERE `resources`.`id` = 117
1063 Query SELECT `param_values`.* FROM `param_values` WHERE (`param_values`.resource_id = 118)
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 700
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 701
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 702
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 703
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 704
1063 Query DELETE FROM `param_values` WHERE `param_values`.`id` = 705
1063 Query SELECT `resource_tags`.* FROM `resource_tags` WHERE (`resource_tags`.resource_id = 118)
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1257
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1258
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1259
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1260
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1261
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1262
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1263
1063 Query DELETE FROM `resource_tags` WHERE `resource_tags`.`id` = 1264
1063 Query DELETE FROM `resources` WHERE `resources`.`id` = 118
1063 Query DELETE FROM `hosts` WHERE `hosts`.`id` = 45
1063 Query COMMIT
1063 Quit
Other than making it obivous as to why storeconfigs is notoursly slow for large installations… It gave me enough of a clue about what was going on make some educated guesses about what is required to purge exported resources without having to nuke all information related to the node. Below is the recipe that I came up with.
-- map the FQDN into the numeric host_id used in the schema SELECT @host_id := id FROM hosts WHERE hosts.name = 'hostname.example.org' LIMIT 1; -- queries to find the records that define all exported resources for the node SELECT * FROM resources r WHERE r.host_id = @host_id and exported = true; SELECT * FROM resource_tags rt join resources r on r.id = rt.resource_id WHERE r.host_id = @host_id and exported = true; SELECT * FROM param_values pv join resources r on r.id = pv.resource_id WHERE r.host_id = @host_id and exported = true; -- purge those rows; note that the order is inverted to not break the joins DELETE pv FROM param_values pv join resources r on r.id = pv.resource_id WHERE r.host_id = @host_id and exported = true; DELETE rt FROM resource_tags rt join resources r on r.id = rt.resource_id WHERE r.host_id = @host_id and exported = true; DELETE r FROM resources r WHERE r.host_id = @host_id and exported = true;
Note that this uses a mysql session variable to simplify the joins but requires that all queries be run in the same session.
This was confirmed to work by kicking off an agent run on a node that had prevously been collecting resources from the node in question.