Drupal

Creating Reusable Dynamic Content Components

admin — Sun, 26 Dec 2021 18:36:24 +0000

Creating Reusable Dynamic Content Components admin Sun, 12/26/2021 - 18:36

November 6, 2018

This is part 2 in this series that explores how to use paragraph bundles to store configuration for dynamic content. The example I built in part 1 was a "read next" section, which could then be added as a component within the flow of the page. The strategy makes sense for component-based sites and landing pages, but probably less so for blogs or content heavy sites, since what we really want is for each article to include the read next section at the end of the page. For that, a view that displays as a block would perfectly suffice. In practice, however, it can be really useful to have a single custom block type, which I often call a "component block", that has an entity reference revisions field that we can leverage to create reusable components.

This strategy offers a simple and unified interface for creating reusable components and adding them to sections of the page. Combined with Pattern Lab and the block visibility groups module, we get a pretty powerful tool for page building and theming.

The image below captures the configuration screen for the "Up next" block you can find at the bottom of this page. As you see, it sets the heading, the primary tag, and the number of items to show. Astute readers might notice, however, that there is a small problem with this implementation. It makes sense if all the articles are about Drupal, but on sites where there are lots of topics, having a reusable component with a hard-coded taxonomy reference makes less sense. Rather, we'd like the related content component to show content that is actually related to the content of the article being read.

For the purpose of this article, let's define the following two requirements: first, if the tagged content component has been added as a paragraph bundle to the page itself, then we will respect the tag supplied in its configuration. If, however, the component is being rendered in the up next block, then we will use the first term the article has been tagged with.

To do that, we need three things: 1) we need our custom block to exist and to have a delta that we can use, 2) we need a preprocess hook to assign the theme variables, and 3) we need a twig template to render the component. If you're following along in your own project, then go ahead and create the component block now. I'll return momentarily to a discussion about custom block and the config system.

Once the up next block exists, we can create the following preprocess function:

function component_helper_preprocess_block__upnextblock(&$variables) {
  if ($current_node = \Drupal::request()->attributes->get('node')) {
    $variables['primary_tag'] = $current_node->field_tags->target_id;
    $variables['nid'] = $current_node->id();
    $paragraph = $variables['content']['field_component_reference'][0]['#paragraph'];
    $variables['limit'] = $paragraph->field_number_of_items->getValue()[0]['value'];
    $variables['heading'] = $paragraph->field_heading->getValue()[0]['value'];
  }
}

If you remember from the first article, our tagged content paragraph template passed those values along to Pattern Lab for rendering. That strategy won't work this time around, though, because theme variables assigned to a block entity, for example, are not passed down to the content that is being rendered within the block.

You might wonder if it's worth dealing with this complexity, given that we could simply render the view as a block, modify the contextual filter, place it and be done with it. What I like about this approach is the flexibility it gives us to render paragraph components in predictable ways. In many sites, we have 5, 10 or more component types. Not all (or even most) of them are likely to be reused in blocks, but it's a nice feature to have if your content strategy requires it. Ultimately, the only reason we're doing this small backflip is because we want to use the article's primary tag as the argument, rather than what was added to the component itself. In other component blocks (an image we want in the sidebar, for example) we could simply allow the default template to render its content.

In the end, our approach is pretty simple: Our up next block template includes the paragraph template, rather than the standard block {{ content }} rendering. This approach makes the template variables we assigned in the preprocess function available:

{% include "@afro_theme/paragraphs/paragraph--tagged-content.html.twig" %}

A different approach to consider would be adding a checkbox to the tagged content configuration, such as "Use page context instead of a specified tag". That would avoid having us having an extra hook and template. Other useful configuration fields we've used for dynamic component configuration include whether the query should require all tags, or any tag, when multiple are assigned, or the ability to specify whether the related content should exclude duplicates (useful when you have several dynamic components on a page but you don't want them to include the same content).

As we wrap up, a final note I'll add is about custom blocks and the config system. The apprach I've been using for content entities that also become config (which is the case here), is to first create the custom block in my local development environment, then export the config and remove the UUID from the config while also copying the plugin uuid. You can then create an update hook that creates the content for the block before it gets imported to config:

/**
 * Adds the "up next" block for posts.
 */
function component_helper_update_8001() {
  $blockEntityManager = \Drupal::service('entity.manager')
    ->getStorage('block_content');

  $block = $blockEntityManager->create(array(
    'type' => 'component_block',
    'uuid' => 'b0dd7f75-a7aa-420f-bc86-eb5778dc3a54',
    'label_display' => 0,
  ));

  $block->info = "Up next block";

  $paragraph = Drupal\paragraphs\Entity\Paragraph::create([
    'type' => 'tagged_content',
    'field_heading' => [
      'value' => 'Up next'
    ],
    'field_number_of_items' => [
      'value' => '3'
    ],
    'field_referenced_tags' => [
      'target_id' => 1,
    ]
  ]);

  $paragraph->save();
  $block->field_component_reference->appendItem($paragraph);
  $block->save();
}

Once we deploy and run the update hook, we're able to import the site config and our custom block should be rendering on the page. Please let me know if you have any questions or feedback in the comments below. Happy Drupaling.

Tags

Drupal

Drupal Pullquotes

admin — Sun, 26 Dec 2021 18:33:36 +0000

Drupal Pullquotes admin Sun, 12/26/2021 - 18:33

October 29, 2018

"Pullquotes", as described here, differ from blockquotes because they duplicate a section of text within the page, and pullquotes get styled in a way that draws the reader's attention to the quote. As such, one simple solution that I've been using is to allow content editors to select a section of text while editing and click a button in the interface to designate it as a pullquote.

TL;DR: Grab your copy on GitHub today.

A quick overview of what's needed

We'll walk through the various pieces required in order to get this working on your site, but it can be summarized in three basic parts:

The module code for the ckeditor plugin
Configuring at least one of your text formatters to include the pullquote button
Adding the js and css to your theme for the site

Adding the Plugin

At this point, I'm not planning to release this as a contrib module for Drupal. If you are interested in doing that, however, then please know you have my full blessing. On projects where pullquotes are required, I tend to simply add them to a small, custom module that includes all of the specific administration tweaks for that project. If you choose to download the GitHub repo, then you can simply drop the "pullquotes" folder into modules/custom/, and drush en pullquotes. If you want to add it to a different custom module, simply update the appropriate paths in src/Plugin/CKEditorPlugin/Pullquote.php.

Configuring a Text Format

The next step is to add the ckeditor button to one of your text formats at /admin/config/content/formats. See the image pictured below.

If the format includes "Limit allowed HTML tags and correct faulty HTML", then be sure to add to the allowed HTML tags, since that is how the plugin modifies the html and prepares it for your theme's javascript. Finally, export your configuration and commit it to your project repo.

Theming the pullquotes

The last step is to modify the javascript and styles that your theme uses to display the pullquotes that have been added in the editing interface. As you can see from the GitHub repo, there are four files that will need to be updated or added to your theme:

your theme info file
your theme library file
the javascript file that adds the markup
the scss (or css) file

In our case, the javascript finds any/all pullquote spans on the page, and then adds them as asides to the DOM, alternating between right and left alignment (for desktop). The scss file then styles them appropriately for small and large breakpoints. Note, too, that the theme css includes specific styles that display in the editing interface, so that content creators can easily see when a pullquote is being added or modified. To remove a pullquote, the editor simply selects it again (which turns the pullquote pink in our theme) and clicks the ckeditor button.

That wraps up this simple tutorial. You can now rest assured that your readers will never miss an important quote again. The strategy is in no way bulletproof, and so its mileage may vary, but if you have questions, feedback, or suggestions on how this strategy can be improved, please add your comment below.

Tags

Drupal

Creating Paragraphs Entities for Dynamic Content

admin — Sun, 26 Dec 2021 18:28:01 +0000

Creating Paragraphs Entities for Dynamic Content admin Sun, 12/26/2021 - 18:28

September 26, 2018

Preamble

I've been working with (and loving) Pattern Lab for the past year and a half, and trying to find time to share some of the most valuable lessons I've learned. This tutorial is the first in what I hope will become a series of articles that demonstrate how I'm solving common requests and requirements for our projects. The underlying concept here does not require Pattern Lab, though I find it a particularly flexible and (mostly) intuitive way to create highly predictable, modular and reusable components.

Background

The paragraphs module has become a central ingredient for many component-based sites in recent years. However, our content strategy also often requires components that display dynamic content (think "Read Next", or "Also of Interest"). In this tutorial, I'll demonstrate how we've been solving this problem, by building paragraph bundles that serve as configuration entities that we can then use as arguments that we pass to a view via the Twig Tweak module. You can see a working version of the dynamic content component we'll be building in the "Up Next" card grid at the bottom of this tutorial.

Creating the View

We want our content editors to be able to add dynamic content to a page that includes a heading, the tag that should be used as a filter, and the number of items they want to display. To start with, let's create a view that is configured as a block of teasers with two contextual filters, the tag (or tags), and a filter that excludes a particular node id. The latter will prevent the current node from displaying within the block. I'm not going to offer a detailed explanation of building views in Drupal, since this is likely quite familiar to most readers.

Building the Paragraph Bundle

Once the view has been configured, it's time to create the paragraph bundle. I'll call it "Tagged content", and add the "configuration" fields we will need in order to render the dynamic content described above.

Once the bundle exists and the fields are present, we can ignore the "Manage display" interface, since we're going to circumvent the render pipeline via Twig Tweak. However, if we add a tagged content bundle to a page (in this case, I already have a content type that allows me to add paragraphs bundles to the page), we should see the values that we set being rendered to the screen via the paragraphs module base template.

The next step, therefore, is creating a paragraph item template that is responsible for rendering the view rather than the actual content of the item, using the "configuration settings" from the paragraph item that we just created.

Twig Field Value and Twig Tweak FTW

Our "configuration" entity is now ready for rendering the view. To do that, we'll leverage the Twig Field Value module to get the values from the paragraph bundle and pass them to the view via Twig Tweak. Here's how that looks in the paragraph item template file (paragraph--tagged-content.html.twig):

{% set tids = content.field_referenced_tags['#items'] is not empty
  ? content.field_referenced_tags|field_raw('target_id')|safe_join(',')
  : 'all' %}

{% set limit = content.field_number_of_items|field_raw('value') %}

{% set heading = content.field_heading|field_raw('value') %}

{# Leverage twig_tweak to create the render array #}
{% set view = drupal_view('tagged_content', 'block_1', tids, nid, limit, heading) %}

{{ view }}

After rebuilding the cache and refreshing the page, we'll now see the content teasers being rendered, though our "Number of items" setting isn't yet working. Also, the value of nid is undefined, which is what we need in order to exclude the current node from the tagged content list. In order to accomplish those two things, as well as prepare for future component hook implementations, we'll create a small custom module called "component helper". The relevant hooks are:

/**
 * Implements hook_preprocess_entity().
 */
function component_helper_preprocess_paragraph__tagged_content(&$variables) {
  if ($current_node = \Drupal::request()->attributes->get('node')) {
    $variables['nid'] = $current_node->id();
  }
}

and

/**
 * Implements hook_views_query_substitutions().
 * Sets the number of items based on the paragraph bundle setting
 * Using this hook instead of alter so that queries can be cached.
 */
function component_helper_views_query_substitutions(ViewExecutable $view) {
  if ($view->id() == 'tagged_content') {
    $limit = (isset($view->args[2])) ? $view->args[2] : 0;
    if ($limit > 0) {
      $view->query->setLimit($limit);
    }
  }
}

Rendering the View Through Pattern Lab

Now we're ready to map our variables in Drupal and send them to be rendered in Pattern Lab. If you're not familiar with it, I suggest you start by learning more about Emulsify, which is Four Kitchens' Pattern Lab-based Drupal 8 theme. Their team is not only super-helpful, they're also very active on the DrupalTwig #pattern-lab channel. In this case, we're going to render the teasers from our view as card molecules that are part of a card grid organism. In order to that, we can simply pass the view rows to the the organism, with a newly created view template (views-view--tagged-content.html.twig):

{# Note that we can simply pass along the arguments we sent via twig_tweak  #} 

{% set heading = view.args.3 %}

{% include '@organisms/card-grid/card-grid.twig' with {
  grid_content: rows,
  grid_blockname: 'card',
  grid_label: heading
} %}

Since the view is set to render teasers, the final step is to create a Drupal theme template for node teasers that will be responsible for mapping the field values to the variables that the card template in Pattern Lab expects.

Generally speaking, for Pattern Lab projects I subscribe to the principle that the role of our Drupal theme templates is to be data mappers, whose responsibility it is to take Drupal field values and map them to Pattern Lab Twig variables for rendering. Therefore, we never output HTML in the theme template files. This helps us keep a clean separation of concerns between Drupal's theme and Pattern Lab, and gives us more predictable markup (note more, since this only applies to templates that we're creating and adding to the theme; otherwise, the Drupal render pipeline is in effect). Here is the teaser template we use to map the values and send them for rendering in Pattern Lab (node--article--teaser.html.twig):

{% set img_src = (img) ? img.uri|image_style('teaser') : null %}

{% include "@molecules/card/01-card.twig" with {
  "card_modifiers": 'grid-item',
  "card_img_src": img_src,
  "card_title": label,
  "card_link_url": url,
} %}

If you're wondering about the img object above, that's related to another custom module that I wrote several years ago to make working with images from media more user friendly. It's definitely out of date, so if you're interested in better approaches to responsive images in Drupal and Pattern Lab, have a look at what Mark Conroy has to say on the topic. Now, if we clear the cache and refresh the page, we should see our teasers rendering as cards (see "Up Next" below for a working version).

Congrats! At this point, you've reached the end of this tutorial. Before signing off, I'll just mention other useful "configuration" settings we've used, such as "any" vs. "all" filtering when using multiple tags, styled "variations" that we can leverage as BEM modifiers, and checkboxes that allow a content creator to specify which content types should be included. The degree of flexibility required will depend on the content strategy for the project, but the underlying methodology works similarly in each case. Also, stay tuned, as in the coming weeks I'll show you how we've chosen to extend this implementation in a way that is both predictable and reusable (blocks, anyone?).

Tags

Drupal

Multiple MailChimp Accounts with Drupal

admin — Sun, 26 Dec 2021 18:25:25 +0000

Multiple MailChimp Accounts with Drupal admin Sun, 12/26/2021 - 18:25

December 20, 2016

A couple of months ago, team ThinkShout quietly introduced a feature to the MailChimp module that some of us have really wanted for a long time—the ability to support multiple MailChimp accounts from a single Drupal installation. This happened, in part, after I reached out to them on behalf of the stakeholders at Cornell University's ILR School, where I work. Email addresses can be coveted resources within organizations, and along with complex governance requirements, it's not uncommon for a single organization to have internal groups who use separate MailChimp accounts. I'm not going to comment on whether this is a good or wise practice, just to acknowledge that it's a reality.

In our case, we currently have three groups within school who are using MailChimp extensively to reach out to their constituents. Up until this week, this required a manual export of new subscribers (whom we are tracking with entityforms), some custom code to transform the csv values into the correct format for MailChimp, and then a manual import to the respective list. However, as our most recent deployment, we are now able to support all three group's needs (including one who is using MailChimp automations). Let's dig into how we're doing it.

The important change that ThinkShout introduced came from this commit, which invokes a new alter hook that allows a developer to modify the key being used for the API object. And though this is an essential hook if we want to enable multiple keys, it also doesn't accomplish much given that mailchimp_get_api_object is called in dozens of places through the suite of MailChimp modules and therefore it's difficult to know the exact context of the api request. For that reason, we really need a more powerful way to understand the context of a given API call.

To that end, we created a new sandbox module called MailChimp Accounts. This module is responsible for five things:

Allowing developers to register additional MailChimp accounts and account keys
Enabling developers to choose the appropriate account for MailChimp configuration tasks
Switching to the correct account when returning to configuration for an existing field or automation entity
Restarting the form rendering process when a field widget needs to be rendered with a different MailChimp account
Altering the key when the configuration of a MailChimp-related field or entity requires it

If you want to try this for yourself, you'll first need to download the newest release of the MailChimp module, if you're not already running it. You'll also need to download the MailChimp Accounts sandbox module. The core functionality of the MailChimp Accounts module relies on its implementation of a hook called hook_mailchimp_accounts_api_key, which allows a module to register one or more MailChimp accounts.

In order to register a key, you will need to find the account id. Since MailChimp doesn't offer an easy way to discover an account id, we built a simple callback page that allows you to retrieve the account data for a given key. You'll find this in the admin interface at /admin/config/mailchimp/account-info on your site. When you first arrive at that page, you should see the account values for your "default" MailChimp account. In this case, "default" simply means it's the API key registered through the standard MailChimp module interface, which stores the value in the variables table. However, if you input a different key on that page, you can retrieve information about that account from the MailChimp API.

The screenshot above offers an example of some of the data that is returned by the MailChimp API, but you will see additional information including list performance when you query an active account. The only essential piece of information, however, is the account_id, which we will use to register additional accounts that we can then use via hook_mailchimp_accounts_api_key(). Here's an example of how to implement that hook:


/**
 * Register API key(s) for MailChimp Accounts
 *
 * @return array
 *   The keys are the account id and the values are the API keys
 */
function mymodule_mailchimp_accounts_api_key() { 
  $keys = array(
    '2dd44aa1db1c924d42c047c96' => 'xxxxxxxx123434234xxxxxx3243xxxx3-us13',
    '411abe81940121a1e89a02abc' => '123434234xxxxxx23233243xxxxxxx13-us12',
  ); 
  return $keys; 
}

Once there is more than one API key registered in a Drupal site, you will see a new option to select the current account to use for any MailChimp configuration taking place in the administrative interface. After selecting a different account, you will also see a notice clarifying which API key is currently active in the admin interface. You can see an example in the screenshot below.

After choosing an account to use for configuration, administrative tasks such as MailChimp subscription fields will use that key when making API calls. Therefore, the available options for lists, merge fields and interest groups will correspond to the appropriate account. Additionally, when the field widget renders the form element, the API key is altered yet again so that the subscriber will be added to the appropriate list, interest groups, etc. Additionally, I can confirm from my testing that the interface for Mailchimp Automations entities also uses the correct API key, enabling support for that MailChimp submodule.

This concludes our walkthrough of the new MailChimp Accounts module. Admittedly, it's not the most elegant solution ever created, but it not only satisfies our organization's complex governance requirements, but it also allows our stakeholders to begin thinking about new possibilities for improving their marketing and communications efforts.

Tags

Drupal

Introducing MailChimp Automations for Drupal

admin — Sun, 26 Dec 2021 18:24:17 +0000

Introducing MailChimp Automations for Drupal admin Sun, 12/26/2021 - 18:24

November 30, 2016

Recent additions to Drupal 7’s MailChimp module and API library offer some powerful new ways for you to integrate Drupal and MailChimp. As of version 7.x-4.7, the Drupal MailChimp module now supports automations, which are incredibly powerful and flexible ways to trigger interactions with your users. Want to reach out to a customer with product recommendations based on their purchase history? Have you ever wished you could automatically send your newsletter to your subscribers when you publish it in Drupal? Or wouldn’t it be nice to be able to send a series of emails to participants leading up to an event without having to think about it? You can easily do all of those things and much more with MailChimp automations.

First of all, big thanks to the team at ThinkShout for all their great feedback and support over the past few months of development. To get started, download the newest release of the MailChimp module and then enable the MailChimp Automations submodule (drush en mailchimp_automations -y). Or, if you’re already using the MailChimp module, update to the latest version, as well as the most recent version of the PHP API library. Also note that although MailChimp offers many powerful features on their free tier, automations are a paid feature with plans starting at $10/month.

Once you follow the readme instructions for the MailChimp module (and have configured your API key), you’re ready to enable your first automation. But before we do that, I’d like to take a step back and talk a bit more conceptually about automations and how we’re going to tie them to Drupal. The MailChimp documentation uses the terms “automations” and “workflows” somewhat interchangeably, which can be a bit confusing at first. Furthermore, any given workflow can have multiple emails associated with it, and you can combine different triggers for different emails in a single workflow. Triggers can be based on a range of criteria, such as activity (clicking on an email), inactivity (failing to open an email), time (a week before a given date), or, as in our case, an integration, which is MailChimp’s term for it since the API call is the trigger.

The api resource we’re using is the automation email queue, which requires a workflow id, a workflow email id, and an email address. In Drupal, we can now accomplish this by creating a MailChimp Automation entity, which is simply a new Drupal entity that ties any other Drupal entity—provided it has an email field—with a specific workflow email in MailChimp. Once you have that simple concept down, the sky’s the limit in terms of how you integrate the entities on your site with MailChimp emails. This means, for example, that you could tie an entityform submission with a workflow email in MailChimp, which could, in turn, trigger another time-based email a week later (drip campaign, anyone?).

This setup becomes even more intriguing when you contrast it to an expensive marketing solution like Pardot. Just because you may be working within tight budget constraints doesn’t mean you can’t have access to powerful, custom engagement tools. Imagine you’re using another one of ThinkShout’s projects, RedHen CRM, to track user engagement on your site. You’ve assigned scoring point values to various actions a user might take on your site, reading an article, sharing it on social media, or submitting a comment. In this scenario, you could track a given user’s score over time, and then trigger an automation when a user crosses a particular scoring threshold, allowing you to even further engage your most active users on the site.

I’m only beginning to scratch the surface on the possibilities of the new MailChimp Automations module. If you're interested in learning more, feel free to take a look at our recent Drupal Camp presentation, or find additional inspiration in both the feature documentation and automation resource guide. Stay tuned, as well, for an upcoming post about another powerful new feature recently introduced into the MailChimp module: support for multiple MailChimp accounts from a single Drupal site!

Tags

Drupal

Speed Up Cache Clearing on Drupal 7

admin — Sun, 26 Dec 2021 18:21:52 +0000

Speed Up Cache Clearing on Drupal 7 admin Sun, 12/26/2021 - 18:21

November 24, 2015

Cache clearing nirvana may be two vsets away

tl;dr If your D7 site uses features or has many entity types, some recent patches to the features module and the entity api module may deliver dramatic performance increases when you clear Drupal's cache. The magic:


    $ drush vset features_rebuild_on_flush FALSE
    $ drush vset entity_rebuild_on_flush FALSE

The Backstory

Given that tedbow is a good friend in our little slice of paradise, aka Ithaca, NY, we decided that we were going to embrace the entityform module on the large Drupal migration I was hired to lead. Fifty-eight entityforms and 420 fields later (even with diligent field re-use), we now see how, in some cases, a pseudo-field system has real benefits, even if it's not the most future-proof solution. As our cache clears became slower and slower (at times taking nearly 10 minutes for a teammate with an older computer), I began to suspect that entityform and/or our extensive reliance on the Drupal field system might be a culprit. Two other corroborating data points were the length of time that feature reverts took when they involved entityforms. Even deployments became a hassle because we had to carefully time them if they required the cache to be cleared, which would make the site unresponsive for logged-in users and cache-cold pages for 5 minutes or more. Clearly, something needed to be done.

Diagnosing

I'm sure there are better ways to handle performance diagnostics (using xDebug, for example), but given the procedural nature of drupal_flush_all_caches it seemed like the devel module would work just fine. I modified the code in Drupal's common.inc file to include the following:


function time_elapsed($comment,$force=FALSE) {
  static $time_elapsed_last = null;
  static $time_elapsed_start = null;

  $unit="s"; $scale=1000000; // output in seconds
  $now = microtime(true);
  if ($time_elapsed_last != null) {
    $elapsed = round(($now - $time_elapsed_last)*1000000)/$scale;
    $total_time = round(($now - $time_elapsed_start)*1000000)/$scale;
    $msg = "$comment: Time elapsed: $elapsed $unit,";
    $msg .= " total time: $total_time $unit";
    dpm($msg);
  }
  else {
      $time_elapsed_start=$now;
  }
  $time_elapsed_last = $now;
}
/**
 * Flushes all cached data on the site.
 *
 * Empties cache tables, rebuilds the menu cache and theme registries, and
 * invokes a hook so that other modules' cache data can be cleared as well.
 */
function drupal_flush_all_caches(){
  // Change query-strings on css/js files to enforce reload for all users.
  time_elapsed('_drupal_flush_css_js');
  _drupal_flush_css_js();
  time_elapsed('registry_rebuild');
  registry_rebuild();
  time_elapsed('drupal_clear_css_cache');
  drupal_clear_css_cache();
  time_elapsed('drupal_clear_js_cache');
  drupal_clear_js_cache();

  // Rebuild the theme data. Note that the module data is rebuilt above, as
  // part of registry_rebuild().
  time_elapsed('system_rebuild_theme_data');
  system_rebuild_theme_data();
  time_elapsed('drupal_theme_rebuild');
  drupal_theme_rebuild();
  time_elapsed('entity_info_cache_clear');
  entity_info_cache_clear();
  time_elapsed('node_types_rebuild');
  node_types_rebuild();
  // node_menu() defines menu items based on node types so it needs to come
  // after node types are rebuilt.
  time_elapsed('menu_rebuild');
  menu_rebuild();

  time_elapsed('actions_synchronize');
  // Synchronize to catch any actions that were added or removed.
  actions_synchronize();

  // Don't clear cache_form - in-progress form submissions may break.
  // Ordered so clearing the page cache will always be the last action.
  $core = array('cache', 'cache_path', 'cache_filter', 'cache_bootstrap', 'cache_page');
  $cache_tables = array_merge(module_invoke_all('flush_caches'), $core);
  foreach ($cache_tables as $table) {
    time_elapsed("clearing $table");
    cache_clear_all('*', $table, TRUE);
  }

  // Rebuild the bootstrap module list. We do this here so that developers
  // can get new hook_boot() implementations registered without having to
  // write a hook_update_N() function.
  _system_update_bootstrap_status();
}

The next time I cleared cache (using admin_menu, since I wanted the dpm messages available), I saw the following:

registry_rebuild: Time elapsed: 0.003464 s, total time: 0.003464 s

drupal_clear_css_cache: Time elapsed: 3.556191 s, total time: 3.559655 s

drupal_clear_js_cache: Time elapsed: 0.001589 s, total time: 3.561244 s

system_rebuild_theme_data: Time elapsed: 0.003462 s, total time: 3.564706 s

drupal_theme_rebuild: Time elapsed: 0.122944 s, total time: 3.68765 s

entity_info_cache_clear: Time elapsed: 0.001606 s, total time: 3.689256 s

node_types_rebuild: Time elapsed: 0.003054 s, total time: 3.69231 s

menu_rebuild: Time elapsed: 0.052984 s, total time: 3.745294 s

actions_synchronize: Time elapsed: 3.334542 s, total time: 7.079836 s

clearing cache_block: Time elapsed: 31.149723 s, total time: 38.229559 s

clearing cache_ctools_css: Time elapsed: 0.00618 s, total time: 38.235739 s

clearing cache_feeds_http: Time elapsed: 0.003292 s, total time: 38.239031 s

clearing cache_field: Time elapsed: 0.006714 s, total time: 38.245745 s

clearing cache_image: Time elapsed: 0.013317 s, total time: 38.259062 s

clearing cache_libraries: Time elapsed: 0.007708 s, total time: 38.26677 s

clearing cache_token: Time elapsed: 0.007837 s, total time: 38.274607 s

clearing cache_views: Time elapsed: 0.006798 s, total time: 38.281405 s

clearing cache_views_data: Time elapsed: 0.008569 s, total time: 38.289974 s

clearing cache: Time elapsed: 0.006926 s, total time: 38.2969 s

clearing cache_path: Time elapsed: 0.009662 s, total time: 38.306562 s

clearing cache_filter: Time elapsed: 0.007552 s, total time: 38.314114 s

clearing cache_bootstrap: Time elapsed: 0.005526 s, total time: 38.31964 s

clearing cache_page: Time elapsed: 0.009511 s, total time: 38.329151 s

hook_flush_caches: total time: 38.348554 s

Every cache cleared.

My initial response was to wonder how and why the cache_block would take so long. Then, however, I noticed line 59 above, which calls module_invoke_all('flush_caches'), which should have been obvious. Also, given that I was just looking for bottlenecks, I modified both module_invoke($module, $hook) in module.inc, as well as the time_elapsed to get the following:


function time_elapsed($comment,$force=FALSE) {
  static $time_elapsed_last = null;
  static $time_elapsed_start = null;
  static $last_action = null; // Stores the last action for the elapsed time message

  $unit="s"; $scale=1000000; // output in seconds
  $now = microtime(true);

  if ($time_elapsed_last != null) {
    $elapsed = round(($now - $time_elapsed_last)*1000000)/$scale;
    if ($elapsed > 1 || $force) {
      $total_time = round(($now - $time_elapsed_start)*1000000)/$scale;
      $msg = ($force)
        ? "$comment: "
        : "$last_action: Time elapsed: $elapsed $unit,";
      $msg .= " total time: $total_time $unit";
      dpm($msg);
    }
  } else {
      $time_elapsed_start=$now;
  }
  $time_elapsed_last = $now;
  $last_action = $comment;
}


/** From module.inc */
function module_invoke_all($hook) {
  $args = func_get_args();
  // Remove $hook from the arguments.
  unset($args[0]);
  $return = array();
  foreach (module_implements($hook) as $module) {
    $function = $module . '_' . $hook;
    if (function_exists($function)) {
      if ($hook == 'flush_caches') {
        time_elapsed($function);
      }
      $result = call_user_func_array($function, $args);
      if (isset($result) && is_array($result)) {
        $return = array_merge_recursive($return, $result);
      }
      elseif (isset($result)) {
        $return[] = $result;
      }
    }
  }

  return $return;
}

The results pointed to the expected culprits:

registry_rebuild: Time elapsed: 4.176781 s, total time: 4.182339 s

menu_rebuild: Time elapsed: 3.367128 s, total time: 7.691533 s

entity_flush_caches: Time elapsed: 22.899951 s, total time: 31.068898 s

features_flush_caches: Time elapsed: 7.656231 s, total time: 39.112933 s

hook_flush_caches: total time: 39.248036 s

Every cache cleared.

After a little digging into the features issue queue, I was delighted to find out that patches had already been committed to both modules (though entity api does not have it in the release yet, so you have to use the dev branch). Two module updates and two vsets later, I got the following results:

registry_rebuild: Time elapsed: 3.645328 s, total time: 3.649398 s

menu_rebuild: Time elapsed: 3.543039 s, total time: 7.378718 s

hook_flush_caches: total time: 8.266036 s

Every cache cleared.

Cache clearing nirvana reached!

Tags

Drupal

Ansible and Drupal Development - Part 2

admin — Sun, 26 Dec 2021 18:20:35 +0000

Ansible and Drupal Development - Part 2 admin Sun, 12/26/2021 - 18:20

November 3, 2014

In part 1 of this tutorial, we covered how to configure and use Ansible for local Drupal development. If you didn't have a chance to read that article, you can download my fork of Jeff Geerling's Drupal Dev VM to see the final, working version from part 1. In this article, we'll be switching things up quite a bit as we take a closer look at the 2nd three requirements, namely:

Using the same playbook for both local dev and remote administration (on DigitalOcean)
Including basic server security
Making deployments simple

TL;DR Feel free to download the final, working version of this repo and/or use it to follow along with the article.

Caveat Emptor

Before we dig in, I want to stress that I am not an expert in server administration and am relying heavily on the Ansible roles created by Jeff Geerling. The steps outlined in this article come from my own experience of trying use Ansible to launch this site, and I'm not aware of how they stray from best-practices. But if you're feeling adventurous, or like me, foolhardy enough to jump in headfirst and just try to figure it out, then read on.

One of the features that makes Ansible so incredibly powerful is to be able to run a given task or playbook across a range of hosts. For example, when the Drupal Security team announced the SQL injection bug now known as "Drupalgeddon", Jeff Geerling wrote a great post about using Ansible to deploy a security fix on many sites. Given that any site that was not updated within 12 hours is now considered compromised, you can easily see what an important role Ansible can play. Ansible is able to connect to any host that is defined in the default inventory file at /etc/ansible/hosts. However, you can also create a project specific inventory file and put it the git repo, which is what we'll do here.

To start with, we'll add a file called "inventory" and put it in the provisioning folder. Inventories are in ini syntax, and basically allow you to define hosts and groups. For now, simply add the following lines to the inventory:

[dev]
yourdomain.dev

The inventory can define hostnames or IP addresses, so 192.168.88.88 (the IP address from the Vagrantfile) would work fine here as well. Personally, I prefer hostnames because I find them easier to organize and track. It will also help us avoid an issues with Ansible commands on the local VirtualBox. With our dev host defined, we are now able to set any required host-specific variables.

Ansible is extremely flexible in how you create and assign variables. For the most part, we'll be using the same variables for all our environments. But a few of them, such as the Drupal domain, ssh port, etc., will be different. Some of these differences are related to the group (such as the ssh port Ansible connects to), while other's are host-specific (such as the Drupal domain). Let's start by creating a folder called "host_vars" in the provisioning folder with a file in it named with the host name of your dev site (a-fro.dev for me). Add the following lines to it:

---
drupal_domain: "yourdomain.dev"

At this point, we're ready to dig into remote server configuration for the first time. Lately, I've been using DigitalOcean to host my virtual servers because they are inexpensive (starting at $5/month) and they have a plethora of good tutorials that helped me work through the manual configuration workflows I was using. I'm sure there are many other good options, but the only requirement is to have a server to which you have root access and have added your public key. I also prefer to have a staging server where I can test things remotely before deploying to production, so for the sake of this tutorial let's create a server that will host stage.yourdomain.com. If you're using a domain for which DNS is not yet configured, you can just add it to your system's hosts file and point to the server's IP address.

Once you've created your server (I chose the most basic plan at DO and added Ubuntu 12.04 x32), you'll want to add it to your inventory like so:

[staging]
stage.yourdomain.com

Assuming that DNS is either already set up, or that you've added the domain to your hosts file, Ansible is now almost ready to talk to the server for the first time. The last thing Ansible needs is some ssh configuration. If you're used to adding this to your ~/.ssh/config file, that's fine. That approach would work fine for now, but we'll see that it will impose some limitations as we move forward, so let's go ahead and add the ssh config to the host file (host_vars/stage.yourdomain.com):

---
drupal_domain: "stage.yourdomain.com"
ansible_ssh_user: root
ansible_ssh_private_key_file: '~/.ssh/id_rsa'

At this point, you should have everything you need to connect to your virtual server and configure it via Ansible. You can test this by heading to the provisioning folder of your repo and typing ansible staging -i inventory -m ping, where "staging" is the group name you defined in your inventory file. You should see something like the following output:

stage.yourdomain.com | success >> {
    "changed": false,
    "ping": "pong"
}

If that's what you see, then congratulations! Ansible has just talked to your server for the first time. If not, you can try running the same command with -vvvv and check the debug messages. We could run the playbook now from part 1 and it should configure the server, but before doing that, let's take a look at the next requirement.

Basic Server Security

Given that the Drupal Dev VM is really set up to support a local environment, it's missing important security features and requirements. Luckily, Jeff comes to the rescue again with a set of additional Ansible roles we can add to the playbook to help fill in the gaps. We'll need the roles installed on our system, which we can do with ansible-galaxy install -r requirements.txt (read more about roles and files). If you already have the roles installed, the easiest way to make sure they're up-to-date is with ansible-galaxy install -r requirements.txt --force (since updating a role is not yet supported by Ansible Galaxy).

In this section, we'll focus on the geerlingguy.firewall and geerlingguy.security roles. Jeff uses the same pattern for all his Ansible roles, so it's easy to find the default vars for a given role by replacing the role name (ie: ansible-role-rolename) of the url: https://github.com/geerlingguy/ansible-role-security/blob/master/defaults/main.yml. The two variables that we care about here are security_ssh_port and security_sudoers_passwordless. This role is going to help us remove password authentication, root login, change the ssh port and add a configured user account to the passwordless sudoers group.

You might notice that the role says "configured user accounts", which begs the question: where does the account get configured? This was actually a stumbling block for me for a while, as I had to work through many different issues my attempts to create and configure the role. The approach we'll take here is working, though may not be the most efficient (or best-pratice, see Caveat Emptor above). Yet there is another issue as well, because the first time we connect to the server it will be over the default ssh port (22), but in the future, we want to choose a more secure port. We're also going to need to make sure that port gets opened on the firewall.

Ansible's variable precendence is going to help us work through these issues. To start with, let's take a look at the following example vars file:

---
ntp_timezone: America/New_York

firewall_allowed_tcp_ports:
  - "{{ security_ssh_port }}"
  - "80"

# The core version you want to use (e.g. 6.x, 7.x, 8.0.x).
# A-fro note: this is slightly deceptive b/c it's really used to check out the correct branch
drupal_core_version: "master"

# The path where Drupal will be downloaded and installed.
drupal_core_path: "/var/www/{{ drupal_domain }}/docroot"

# Your drupal site's domain name (e.g. 'example.com').
# drupal_domain:  moved to group_vars

# Your Drupal site name.
drupal_site_name: "Aaron Froehlich's Blog"
drupal_admin_name: admin
drupal_admin_password: password

# The webserver you're running (e.g. 'apache2', 'httpd', 'nginx').
drupal_webserver_daemon: apache2

# Drupal MySQL database username and password.
drupal_mysql_user: drupal
drupal_mysql_password: password
drupal_mysql_database: drupal

# The Drupal git url from which Drupal will be cloned.
drupal_repo_url: "git@github.com:a-fro/a-fro.com.git"

# The Drupal install profile to be used
drupal_install_profile: standard

# Security specific
# deploy_user: defined in group_vars for ad-hoc commands
# security_ssh_port: defined in host_vars and group_vars
security_sudoers_passwordless:
  - "{{ deploy_user }}"
security_autoupdate_enabled: true

You'll notice that some of the variables have been moved to host_vars or group_vars files. Our deploy_user, for example, would work just fine for our playbook if we define it here. But since we want to make this user available to Ansible for ad-hoc commands (not in playbooks), it is better to put it in group_vars. This is also why we can't just use our ~/.ssh/config file. With Ansible, any variables added to provisioning/group_vars/all are made available by default to all hosts in the inventory, so create that file and add the following lines to it:

---
deploy_user: deploy

For the security_ssh_port, we'll be connecting to our dev environment over the default port 22, but changing the port on our remote servers. I say servers (plural), because eventually we'll have both staging and production environments. We can modify our inventory file to make this a bit easier:

[dev]
a-fro.dev

[staging]
stage.a-fro.com

[production]
a-fro.com

[droplets:children]
staging
production

This allows us to issue commands to a single host, or to all our droplets. Therefore, we can add a file called "droplets" to the group_vars folder and add the group-specific variables there:

---
ansible_ssh_user: "{{ deploy_user }}"

security_ssh_port: 4895 # Or whatever you choose
ansible_ssh_port: "{{ security_ssh_port }}"

ansible_ssh_private_key_file: ~/.ssh/id_rsa # The private key that pairs to the public key on your remote server.

Configuring the Deploy User

There are two additional issues that we need to address if we want our security setup to work. The first is pragmatic: using a string in the security_sudoers_passwordless yaml array above works fine, but Ansible throws an error when we try to use a variable there. I have a pull request issued to Ansible-role-security that resolves this issue, but unless that gets accepted, we can't use the role as is. The easy alternative is to download that role to our local system and add it's contents to a folder named "roles" in provisioning (ie. provisioning/roles/security). You can see the change we need to make to the task here. Then, we modify the playbook to use our local "security" role, rather than geerlingguy.security.

The second issue we face is that the first time we connect to our server, we'll do it as root over port 22, so that we can add the deploy_user account, and update the security configuration. Initially, I was just modifying the variables depending on whether it was the first time I was running the playbook, but that got old really quickly as I created, configured and destroyed my droplets to work through all the issues. And while there may be better ways to do this, what worked for me was to add an additional playbook that handles our initial configuration. So create a provisioning/deploy_config.yml file and add the following lines to it:

---

- hosts: all
  sudo: yes

  vars_files:
    - vars/main.yml
    - vars/deploy_config.yml

  pre_tasks:
    - include: tasks/deploy_user.yml

  roles:
    - security

Here's the task that configures the deploy_user:

---
- name: Ensure admin group exists.
  group: name=admin state=present

- name: Add deployment user
  user: name='{{ deploy_user }}'
        state=present
        groups="sudo,admin"
        shell=/bin/bash

- name: Create .ssh folder with correct permissions.
  file: >
    path="/home/{{ deploy_user }}/.ssh/"
    state=directory
    owner="{{ deploy_user }}"
    group=admin
    mode=700

- name: Add authorized deploy key
  authorized_key: user="{{ deploy_user }}"
                  key="{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
                  path="/home/{{ deploy_user }}/.ssh/authorized_keys"
                  manage_dir=no
  remote_user: "{{ deploy_user }}"

The private/public key pair you define in the "Add authorized deploy key" task and in your ansible_ssh_private_key_file variable should have access to both your remote server and your GitHub repository. If you've forked or cloned my version, then you will definitely need to modify the keys.

Our final security configuration prep step is to leverage Ansible's variable precendence to override the ssh settings to use root and the default ssh port with the following lines in provisioning/vars/deploy_config:

---
ansible_ssh_user: root
ansible_ssh_port: 22

We now have everything in place to configure the basic security we're adding to our server. Remembering that one of we want our playbooks to work both locally over Vagrant and remotely, we can first try to run this playbook in our dev environment. I couldn't find a good way to make this seamless with Vagrant, so I've added a conditional statement to the Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at vagrantup.com.

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "ubuntu-precise-64"

  # The url from where the 'config.vm.box' box will be fetched if it
  # doesn't already exist on the user's system.
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  config.vm.network :private_network, ip: "192.168.88.88"

  # Share an additional folder to the guest VM. The first argument is
  # the path on the host to the actual folder. The second argument is
  # the path on the guest to mount the folder. And the optional third
  # argument is a set of non-required options.
  config.vm.synced_folder "../a-fro.dev", "/var/www/a-fro.dev", :nfs => true

  # Configure VirtualBox.
  config.vm.provider :virtualbox do |vb|
    # Set the RAM for this VM to 512M.
    vb.customize ["modifyvm", :id, "--memory", "512"]
    vb.customize ["modifyvm", :id, "--name", "a-fro.dev"]
  end

  # Enable provisioning with Ansible.
  config.vm.provision "ansible" do |ansible|
    ansible.inventory_path = "provisioning/inventory"
    ansible.sudo = true
    # ansible.raw_arguments = ['-vvvv']
    ansible.sudo = true
    ansible.limit = 'dev'

    initialized = false

    if initialized
      play = 'playbook'
      ansible.extra_vars = { ansible_ssh_private_key_file: '~/.ssh/ikon' }
    else
      play = 'deploy_config'
      ansible.extra_vars = {
        ansible_ssh_user: 'vagrant',
        ansible_ssh_private_key_file: '~/.vagrant.d/insecure_private_key'
      }
    end
    ansible.playbook = "provisioning/#{play}.yml"
  end
end

The first time we run vagrant up, if initialized is set to false, then it's going to run deploy_config. Once it's been initialized the first time (assuming there were no errors), you can set initialized to true and from that point on, playbook.yml will run when we vagrant provision. Assuming everything worked for you, then we're ready to configure our remote server with ansible-playbook provisioning/deploy_config.yml -i provisioning/inventory --limit=staging.

Installing Drupal

Whew! Take a deep breath, because we're really at the home stretch now. In part 1, we used a modified Drupal task file to install Drupal. Since then, however, Jeff has accepted a couple of pull requests that get us really close to being able to use his Drupal Ansible Role straight out of the box. I have another pull request issued that get's us 99% of the way there, but since that hasn't been accepted, we're going to follow the strategy we used with the security role and add a "drupal" folder to the roles.

I've uploaded a branch of ansible-role-drupal, that includes the modifications we need. They're all in the provisioning/drupal.yml task, and I've outlined the changes and reasons in my pull request. If you're following along, I suggest downloading that branch from GitHub and adding it to a drupal folder in your provisioning/roles. One additional change that I have not created a pull request for relates to the structure I use for Drupal projects. I like to put Drupal in a subfolder of the repository root (typically called docroot). As many readers will realize, this is in large part because we often host on Acquia. And while we're not doing that in this case, I still find it convenient to be able to add other folders (docs, bin scripts, etc.) alongside the Drupal docroot. The final modification we make, then, is to checkout the repository to /var/www/{{ drupal_domain }} (rather than {{ drupal_core_path }}, which points to the docroot folder of the repo).

We now have all our drops in a row and we're ready to run our playbook to do the rest of the server configuration and install Drupal! As I mentioned above, we can modify our Vagrantfile to set initialized to true and run vagrant provision, and our provisioner should run. If you run into issues, you can uncomment the ansible.raw_arguments line and enable verbose output.

One final note before we provision our staging server. While vagrant provision works just fine, I think I've made my preference clear for having consistency between environments. We can do that here by modifying the host_vars for dev:

---
drupal_domain: "a-fro.dev"
security_ssh_port: 22
ansible_ssh_port: "{{ security_ssh_port }}"
ansible_ssh_user: "{{ deploy_user }}"
ansible_ssh_private_key_file: '~/.ssh/id_rsa'

Now, assuming that you already ran vagrant up with initialized set to false, then you can run your playbook for dev in the same way you will for your remote servers:

cd provisioning
ansible-playbook playbook.yml -i inventory --limit=dev

If everything runs without a hitch on your vagrant server, then you're ready to run it remotely with ansible-playbook playbook.yml -i inventory --limit=staging. A couple of minutes later, you should see your Drupal site installed on your remote server.

Simple Deployments

I'm probably not the only reader of Jeff's awesome book Ansible for Devops who is looking forward to him completing Chapter 9, Deployments with Ansible. In the meantime, however, we can create a simple deploy playbook with two tasks:

---
- hosts: all

  vars_files:
    - vars/main.yml

  tasks:
    - name: Check out the repository.
      git: >
        repo='git@github.com:a-fro/a-fro.com.git'
        version='master'
        accept_hostkey=yes
        dest=/var/www/{{ drupal_domain }}
      sudo: no

    - name: Clear cache on D8
      command:
        chdir={{ drupal_core_path }}
        drush cr
      when: drupal_major_version == 8

    - name: Clear cache on D6/7
      command:
        chdir={{ drupal_core_path }}
        drush cc all
      when: drupal_major_version < 8

Notice that we've added a conditional that checks for a variable called drupal_major_version, so you should add that to your provisioniong/vars/main.yml file. If I was running a D7 site, I'd probably add tasks to the deploy script such as drush fr-all -y, but this suffices for now. Since I'm pretty new to D8, if you have ideas on other tasks that would be helpful (such as a git workflow for CM), then I'm all ears!

Conclusion

I hope you enjoyed this 2 part series on Drupal and Ansible. One final note for the diligent reader relates to choosing the most basic hosting plan, which limits my server to 512MB of ram. I've therefore added an additional task that adds and configures swap space when not on Vagrant.

Thanks to the many committed open source developers (and Jeff Geerling in particular), devops for Drupal are getting dramatically simpler. As the community still reels from the effects of Drupalgeddon, it's easy to see how incredibly valuable it is to be able to easily run commands across a range of servers and codebases. Please let me know if you have questions, issues, tips or tricks, and as always, thanks for reading.

Tags

Drupal

Keeping Compiled CSS Out of your Git Repository on Acquia [Updated]

admin — Sun, 26 Dec 2021 18:17:28 +0000

Keeping Compiled CSS Out of your Git Repository on Acquia [Updated] admin Sun, 12/26/2021 - 18:17

October 27, 2014

A couple of months ago, after a harrowing cascade of git merge conflicts involving compiled css, we decided it was time to subscribe to the philosophy that compiled CSS doesn't belong in a git repository. Sure, there are other technical solutions teams are tossing around that try to handle merging more gracefully, but I was more intererested in simply keeping the CSS out of the repo in the first place. After removing the CSS from the repo, we suddenly faced two primary technical challenges:

During development, switching branches will now need to trigger a recompliation of the stylesheets
Without the CSS in the repo, it's hard to know how to get the code up to Acquia

In this article, I'll describe the solutions we came up with to handle these challenges, and welcome feedback if you have a different solution.

Local Development

If you're new to using tools like Sass, Compass, Guard and LiveReload, I recommend taking a look at a project like Drupal Streamline. For the purpose of this post, I'm going to assume that you're already using Compass in your project. Once the CSS files have been removed, you'll want to compass compile to trigger an initial compilation of the stylesheet. However, having to remember to compile every time you switch to a new branch introduces not only an inconvenience, but also a strong possiblily for human error.

Luckily, we can use git hooks to remove this risk and annoyance. In this case, we'll create a post-checkout hook that triggers compiling every time a new branch is checked out:

Create a file called post-checkout in the .git/hooks folder

Add the following lines to that file:

#! /bin/sh
# Start from the repository root.
cd ./$(git rev-parse --show-cdup)
compass compile

From the command line in the repository root, type chmod +x .git/hooks/post-checkout

Assuming you have compass correctly configured, you should see the stylesheets getting re-compiled the next time you git checkout [branch], even if you're not already running Guard and LiveReload.

Deploying to Acquia

Now that CSS is no longer being deployed when we push our repo up to Acquia, we need to figure out how we're going to get it there. It would be possible to force-add the ignored stylesheets before I push the branch up, but I don't really want all those additional commits on my development branches in particular. Luckily, Acquia has a solution that we can hack which will allow us to push the files up to Dev and Stage (note, we'll handle prod differently).

Enter LiveDev

Acquia has a setting that you can toggle on both the dev and test environments that allows you to modify the files on the server. It's called 'livedev', and we're going to exploit its functionality to get our compiled CSS up to those environments. After enabling livedev in the Acquia workflow interface, you are now able to scp files up to the server during deployment. Because I like to reduce the possibility of human error, I prefer to create a deploy script that handles this part for me. It's basically going to do three things:

Compile the css
scp the css files up to Acquia livedev for the correct environment
ssh into Acquia's server and checkout the code branch that we just pushed up.

Here's the basic deploy script that we can use to accomplish these goals:

#!/bin/bash

REPO_BASE='[project foldername here (the folder above docroot)]'

# check running from the repository base
CURRENT_DIR=${PWD##*/}
if [ ! "$CURRENT_DIR" = $REPO_BASE ]; then
  echo 'Please be sure that you are running this command from the root of the repo.'
  exit 2
fi

# Figure out which environment to deploy to
while getopts "e:" var; do
    case $var in
        e) ENV="${OPTARG}";;
    esac
done

# Set the ENV to dev if 'e' wasn't passed as an argument
if [ "${#ENV}" -eq "0" ]; then
  ENV='dev'
fi

if [ "$ENV" = "dev" ] || [ "$ENV" = "test" ]; then
  # Set the css_path and livedev path
  CSS_PATH='docroot/sites/all/themes/theme_name/css/'

  # Replace [user@devcloud.host] with your real Acquia Cloud SSH host
  # Available in the AC interface under the "Users and keys" tab
  ACQUIA_LIVEDEV='[user@devcloud.host]:~/$ENV/livedev/'

  # Get the branch name
  BRANCH_NAME="$(git symbolic-ref HEAD 2>/dev/null)" ||
  BRANCH_NAME="detached"     # detached HEAD
  BRANCH_NAME=${BRANCH_NAME##refs/heads/}

  echo "Pushing $BRANCH_NAME to acquia cloud $ENV"
  git push -f ac $BRANCH_NAME # This assumes you have a git remote called "ac" that points to Acquia

  echo "Compiling css"
  compass compile

  # Upload to server
  echo "Uploading styles to server"
  scp -r $CSS_PATH "$ACQUIA_LIVEDEV~/$ENV/livedev/$CSS_PATH":

  # Pull the updates from the branch to livedev and clear cache
  echo "Deploying $BRANCH_NAME to livedev on Acquia"
  ssh $ACQUIA_LIVEDEV "git checkout .; git pull; git checkout $BRANCH_NAME; cd docroot; exit;"

  echo "Clearing cache on $ENV"
  cd docroot
  drush [DRUSH_ALIAS].$ENV cc all -y

  echo "Deployment complete"
  exit
fi

# If not dev or test, throw an error
echo 'Error: the deploy script is for the Acquia dev and test environments'

Now I don't pretend to be a shell scripting expert and I'm sure this script could be improved; however, it might be helpful to explain a few things. To start with, you will need to chmod +x [path/to/file]. I always put scripts like this in a bin folder at the root of the repo. There are a few other variables that you'll need to change if you want to use this script, such as REPO_BASE, CSS_PATH and ACQUIA_LIVEDEV. Also, the script assumes that you have a git remote called "ac", which should point to your Acquia Cloud instance. Finally, the drush cache clear portion assumes that you have a custom drush alias created for your livedev environment for both dev and test; if not, you can remove those lines. To deploy the site to dev, you would run the command bin/deploy, or bin/deploy -e test to deploy to the staging environment.

Deploying to Prod

Wisely, Acquia doesn't provide keys to run livedev on the production environment, and this approach is probably more fragile than we'd like anyway. For the production environment, we're going to use an approach that force-adds the stylesheet when necessary.

To do this, we're again going to rely on a git hook to help reduce the possibility of human error. Because our development philosophy relies on a single branch called "production" that we merge into and tag, we can use git's post-merge hook to handle the necessary force-adding of our stylesheet.

#! /bin/sh

BRANCH_NAME="$(git symbolic-ref HEAD 2>/dev/null)" ||
BRANCH_NAME="detached"
BRANCH_NAME=${BRANCH_NAME##refs/heads/}
CSS_PATH="docroot/sites/all/themes/theme_name/css/"

if [ "$BRANCH_NAME" = "production" ]; then
  compass compile
  git add $CSS_PATH -f
  git diff --cached --exit-code > /dev/null
  if [ "$?" -eq 1 ]; then
    git commit -m 'Adding compiled css to production'
  fi
fi

As with the post-checkout hook, you'll need to make sure this file is executable. Note that after the script stages the css files, git is able to confirm whether there are differences in the current state of the files, and only commit the files when there are changes. After merging a feature branch into the production branch, the post-merge hook gets triggered, and I can then add a git tag, push the code and new tag to the Acquia remote, and then utilize Acquia's cloud interface to deploy the new tag.

Conclusion

While this may seem like a lot of hoops to jump through to keep compiled CSS out of the repository, the deploy script actually fits very nicely with my development workflow, because it allows me to easily push up the current branch to dev for acceptance testing. In the future, I'd like to rework this process to utilize Acquia's Cloud API, but frankly, my tests with the API thus far have returned unexpected results, and I haven't wanted to submit one of our coveted support tickets to figure out why the API isn't working correctly. If you're reading this and can offer tips for improving what's here, sharing how you accomplish the same thing, or happen to work at Acquia and want to talk about the bugs I'm seeing in the API, please leave a comment. And thanks for reading!

Update

Dave Reid made a comment below about alternatives to LiveDev and the possibility of using tags to accomplish this. As I mentioned above, LiveDev works well for me (on dev and test) because it fits well into my typical deployment workflow. The problem I see with using tags to trigger a hook is that we are in the practice of tagging production releases, but not for dev or test. Thinking through Dave's suggestion, however, led to me to an alternative approach to LiveDev that still keeps the repo clean using Git's "pre-push" hook:

#! /bin/sh

PUSH_REMOTE=$1
ACQUIA_REMOTE='ac' #put your Acquia remote name here

if [ $PUSH_REMOTE = $ACQUIA_REMOTE ]; then
  compass compile
  git add docroot/sites/all/themes/ilr_theme/css/ -f
  git diff --cached --exit-code > /dev/null
  if [ "$?" -eq 1 ]; then
    git commit -m "Adding compiled css"
  fi
fi

The hook receives the remote as the first argument, which allows us to check whether we're pushing to our defined Acquia remote. If we are, the script then checks for CSS changes, and adds the additional commit if necessary. The thing I really like about this approach is that the GitHub repository won't get cluttered with the extra commit, but the CSS files can be deployed to Acquia without livedev.

Tags

Drupal

Ansible and Drupal Development

admin — Sun, 26 Dec 2021 18:15:24 +0000

Ansible and Drupal Development admin Sun, 12/26/2021 - 18:15

October 22, 2014

As I mentioned in my hello world post, I've been learning Ansible via Jeff Geerling's great book Ansible for Devops. When learning new technologies, there is no substitute for diving in and playing with them on a real project. This blog is, in part, the byproduct of my efforts to learn and play with Ansible. Yet embedded within that larger goal were a number of additional technical requirements that were important to me, including:

Setting up a local development environment using Vagrant
Installing Drupal from a github repo
Configuring Vagrant to run said repo over NFS (for ST3, LiveReload, Sass, etc.)
Using the same playbook for both local dev and remote administration (on DigitalOcean)
Including basic server security
Making deployments simple

In this blog entry, we'll look at the first three requirements in greater detail, and save the latter three for another post.

Configuring Local Development with Vagrant

At first glance, requirement #1 seems pretty simple. Ansible plays nicely with Vagrant, so if all you want to do is quickly spin up a Drupal site, download Jeff's Drupal Dev VM and you'll be up and running in a matter of minutes. However, when taken in the context of the 2nd and 3rd requirements, we're going to need to make some modifications to the Drupal Dev VM.

To start with, the Drupal Dev VM uses a drush make file to build the site. Since we want to build the site based on our own git repository, we're going to need to find a different strategy. This is actually a recent modification to the Drupal Dev VM, which previously used an Ansible role called "Drupal". If you look carefully at that github repo, you'll actually notice that Jeff accepted one of my pull requests to add the functionality we're looking for from this role. The last variable is called drupal_repo_url, which you can use if you want to install Drupal from your own repository rather than Drupal.org. We'll take a closer look at this in a moment.

Installing Drupal with a Custom Git Repo

Heading back to the Drupal Dev VM, you can see that the principle change Jeff made was to remove geerlingguy.drupal from the dependency list, and replace it with a new task defined in the drupal.yml file. After cloning the Dev VM onto your system, remove - include: tasks/drupal.yml from the tasks section and add - geerlingguy.drupal to the roles section.

After replacing the Drupal task with the Ansible Drupal role, we also need to update the vars file in the local repo with the role-specific vars. There, you can update the drupal_repo_url to point to your github url rather than the project url at git.drupal.org.

Configuring Vagrant for NFS

At this point, we would be able to meet the first two requirements with a simple vagrant up, which would provision the site using Ansible (assuming that you've already installed the dependencies). Go ahead and try it if you're following along on your local machine. But there's a problem, because our third requirement is going to complicate this setup. Currently, Drupal gets downloaded and installed on the VM, which complicates our ability to edit the files using our IDE of choice and also being able to run the necessary Ruby gems like Sass and LiveReload.

When I was initially working through this process, I spent quite a few hours trying to configure my VM to download the necessary Ruby gems so I could compile my stylesheets with Compass directly on the VM. The biggest drawback for me, however, was that I didn't really want to edit my code using Vim over ssh. What I really needed was to be able to share my local git repo of the site with my Vagrant box via NFS, hence the 3rd requirement.

In order to satisfy this 3rd requirement, I ended up removing my dependency on the Ansible Drupal role and instead focussed on modifying the Drupal task to meet my needs. Take a look at this gist to see what I did.

Most of the tasks in that file should be pretty self-explanatory. The only one that might be suprising is the "Copy the css files" task, which is necessary because I like to keep my compiled CSS files out of the repo (more on this coming soon). Here's a gist of an example vars file you could use to support this task.

One other advantage of our modified Drupal task is that we can now specify an install profile to use when installing Drupal. I currently have a pull request that would add this functionality to the Ansible Drupal Role, but even if that gets committed, it won't solve our problem here because we're not using that role. We could, however, simply modify the "Install Drupal (standard profile) with drush" to install our custom profile if that's part of your typical workflow. If I were installing a D7 site here, I would definitely use a custom profile, since that is my standard workflow, but since we're installing D8 and I haven't used D8 profiles yet, I'm leaving it out for now.

The next step we need to take in order to get our site working correctly is to modify the Vagrantfile so that we share our local site. You might have noticed in the vars file that the drupal_css_path variable points to a folder on my system named "a-fro.dev", which is, not suprisingly, the folder we want to load over NFS. This can be accomplished by adding the following line to the Vagrantfile:

config.vm.synced_folder "../a-fro.dev", "/var/www/a-fro.dev", :nfs => true

Note that the folder we point to in /var/www should match the {{ drupal_domain }} variable we previously declared. However, since this is now pointing to a folder on our local system (rather than on the vm), we'll run into a couple of issues when Ansible provisions the VM. Vagrant expects the synced_folder to exist, and will throw an error if it does not. Therefore, you need to make sure an point to an existing folder that includes the path specified in {{ drupal_core_path }}. Alternatively, you could clone a-fro.com repo into the folder above your drupal-dev-vm folder using the command git clone git@github.com:a-fro/a-fro.com.git a-fro.dev. Additionally, you will probably receive an error when the www.yml task tries to set permissions on the www folder. The final change we need to make, then, is to remove the "Set permissions on /var/www" task from provisioning/tasks/www.yml

With this final change in place, we should now be able to run vagrant up and the the site should install correctly. If it doesn't work for you, one possible gotcha is with the task that checks if Drupal is already installed. That task looks for the settings.php file, and if it finds it, the Drush site-install task doesn't run. If you're working from a previously installed local site, the settings.php file may already exist.

Conclusion

This completes our first three requirements, and should get you far enough that you could begin working on building your own local site and getting it ready to deploy to your new server. You can find the final working version from this post on GitHub. In the next blog post, we'll look more closely at the last three requirements, which I had to tackle in order to get the site up and running. Thanks for reading.

Tags

Drupal

Hello World from Drupal 8

admin — Sun, 26 Dec 2021 18:13:24 +0000

Hello World from Drupal 8 admin Sun, 12/26/2021 - 18:13

October 8, 2014

Welcome! This site has been a while in the making, but I'm really excited to share it with you. Back in Austin at DrupalCon, I was inspired by Jeff Geerling's "Devops for Humans" presentation and immediately decided that I needed to start using Ansible. Well, it's been a long road, but the site is now live and I'm really looking forward to sharing the ups and downs of the journey. Oh, and if you don't have it already, Jeff's book Ansible for Devops is well worth it. More soon...

Tags

Drupal

Drupal

Creating Reusable Dynamic Content Components

Drupal Pullquotes

A quick overview of what's needed

Adding the Plugin

Configuring a Text Format

Theming the pullquotes

Creating Paragraphs Entities for Dynamic Content

Preamble

Background

Creating the View

Building the Paragraph Bundle

Twig Field Value and Twig Tweak FTW

Rendering the View Through Pattern Lab

Multiple MailChimp Accounts with Drupal

Introducing MailChimp Automations for Drupal

Speed Up Cache Clearing on Drupal 7

Cache clearing nirvana may be two vsets away

The Backstory

Diagnosing

Ansible and Drupal Development - Part 2

Caveat Emptor

Sharing Playbooks Between Local and Remote Environments

Basic Server Security

Configuring the Deploy User

Installing Drupal

Simple Deployments

Conclusion

Keeping Compiled CSS Out of your Git Repository on Acquia [Updated]

Local Development

Deploying to Acquia

Enter LiveDev

Deploying to Prod

Conclusion

Update

Ansible and Drupal Development

Configuring Local Development with Vagrant

Installing Drupal with a Custom Git Repo

Configuring Vagrant for NFS

Conclusion

Hello World from Drupal 8