The current puppet passanger crypto spec is 5yrs old (talked to the original author). I've adapted it to something sane and have taken the current recommendations for it by the bettercrypto(.org) project.

I profiled puppet agent --test on one of my servers using stackprof [1] on Ruby 2.1.

Something like 30% or more of the time was spent in Puppet::Graph::SimpleGraph#upstream_from_vertex, called from Puppet::Transaction#failed_dependencies?

It turns out that, while evaluating the resource graph, when considering a resource, we look at the complete set of transitive dependencies to evaluate whether any of them have failed. This is hugely expensive, and moreover, is wasted work in the success case where no resources fail.

Instead of all that work, this patch pushes the work to the failed nodes; When a node fails, we transitively walk the dependents of that node, and mark them as having failed dependencies, and then check that flag directly when considering whether to skip a node later.

On my test system, this patch drops puppet agent --test runtime from about 40s to about 23s.

[1] https://github.com/tmm1/stackprof

Under some circumstances (eg. dependency cycle) when catalog could not be applied, Puppet didn't report failure (status unchanged, return code 0).

Raise Puppet::Error so the failure gets logged and reported (status is "failed", return value is 1).

On occasion, a puppet agent can end up waiting indefinitely on some process that will never return or terminate. This patch adds a new setting, runtimeout, that can be used to specify the maximum duration allowed for a puppet run. When set to a non-zero value, the agent daemon will send a SIGTERM to any agent run it starts which exceeds this value.

This patch does not handle runs started outside of the agent daemon by puppet agent --onetime.

This commit adds a new Object type to the Puppet Type system. The type is very similar to Struct in that it defines a hash of named types. There are some notable differences though:

1. The Object type adds inheritance.
2. An Object is only assignable to itself or to a parent type of itself.
3. All keys of the Object must be non empty strings.
4. An Object may redefine a parent member, but only if the type is equal or assignable to the type of that member. I.e. it may redefine it to a more special type.

I'm not familiar with why this causes damaged YAML output, but reverting this performance optimization commit repairs the issue for me with 3.0.0, 3.2.4, and 3.3.0-rc.

See also http://projects.puppetlabs.com/issues/4506#note-22 for details about how this commit was identified.

The comment field for user resources can often contain non-ASCII characters. Unfortunately, Puppet retrieves the existing value (current_value) as ASCII-8BIT, but strings containing non-ASCII characters are automatically turned into UTF-8, resulting in a failure to concatenate the strings in change_to_s, with messages such as the following in Puppet 3.2.4 with Ruby 1.9.3:

Error: Could not convert change 'comment' to string: incompatible character encodings: ASCII-8BIT and UTF-8
Error: Could not convert change 'comment' to string: incompatible character encodings: ASCII-8BIT and UTF-8
Error: /User[rpinson]: Could not evaluate: Puppet::Util::Log requires a message

which is due to the fact that current_value is encoded in ASCII-8BIT, but newvalue is in UTF-8 and not "softly" convertible when reaching change_to_s in property.rb.

In order to allow proper management of non-ASCII characters in the comment field, this PR forces an ASCII-8BIT encoding on the comment property value in the type.

Note: trying to force the encoding in Puppet code (using e.g. inline_template) doesn't solve anything, and patching change_to_s to force the encoding results in the resource not being idempotent anymore, since comparison fails (but change_to_s doesn't crash anymore).

If lookup_options is found in Hiera levels that utilise top scope variables as opposed to facts then unless all top scope variables are set before the first lookup, lookup_options gets cached once and is never refreshed.

This change removes all caching so lookup_options is refreshed every time.

If a Ruby type is not available in the file system, every time it is found by the parser, the Autoloader will scan all over again the directories in search_directories for it. This generates tons of stat()s that can be saved.

This patch annotates that a type could not be found so the next time it is required during the compilation of a catalog, the file system is not hammered again unnecessarily.

This optimization has given us a ~61% reduction in the number of stats per compilation, as we make an extensive use of the concat module which only provides Puppet "defined types".

This adds 2 features:

csr_attributes_file (#7243)

This adds a csr_attributes_file parameter to the puppet config which allows for specifying user defined attributes to be added to the CSR. The file is yaml instead of flat key/value as this allows you to have an array of values for a parameter (which is also allowed in a CSR). However if the user were to provide a hash in the yaml, or a nested array, it would probably break.

Example file:

---
1.3.6.1.4.1.34380.2.0: us-west-1a/i-355fb16d
1.3.6.1.4.1.34380.2.1: MYSUPERSECRETKEY
1.3.6.1.4.1.34380.3.3: puppet-dashboard-group-name

autosign_command (#7244)

This adds a autosign_command parameter to the puppet config which allows for running an external command to determine whether a certificate should be automatically signed or not. The command is passed the certificate name as the first argument. The command should then exit with status 0 to indicate that the certificate should be signed, and status non-0 to indicate it should not be signed.

I use these 2 features for autosigning CSRs that come from EC2 autoscaled instances. Each instance runs a small userdata script which adds a secret key and the name of a dashboard group to the CSR. The autosigning script then looks for the secret to verify the box is indeed mine, adds the box to the requested group in the puppet dashboard, and then exits with 0 to indicate puppet is OK to sign the cert.

I have not had a chance to develop any tests for these changes. It has been on my todo list, but I have received numerous requests to create this pull request. Should I get time to create the tests before anyone else I will do so, but I do not know when that would be.

If a Ruby type is not available in the file system, every time it is found by the parser, the Autoloader will scan all over again the directories in search_directories for it. This generates tons of stat()s that can be saved.

This patch annotates that a type could not be found so the next time it is required during the compilation of a catalog, the file system is not hammered again unnecessarily.

This optimization has given us a ~61% reduction in the number of stats per compilation, as we make an extensive use of the concat module which only provides Puppet "defined types".

Specs generated a warning due to an issue with vcr < 6.1 on Ruby 3.1+. See https://github.com/vcr/vcr/commit/98cc00b57369dfd671c227be6aa3f03d7475cb6c

Early return behavior is already tested in spec/unit/functions/return_spec.rb

We test puppet apply basics and file checksum in multiple places.

Also remove apply basics from Rakefile task

This line is not sane for the setting of the log_level default

loglevel = %x{ #{puppet_path} config --section agent --log_level notice print log_level }.chomp

this sets log_level to notice, and then return the output of the log_level setting

  loglevel = "notice"

is the same thing with fewer steps

Not locking the default initialization can lead to race-conditions.

Note: not sure if I should use one or two mutexes as I am not familiar with this code enough to make the judgment.

ref: https://github.com/ruby-concurrency/concurrent-ruby/issues/970