{"id":7651,"date":"2024-01-29T09:32:20","date_gmt":"2024-01-29T15:32:20","guid":{"rendered":"https:\/\/gregsowell.com\/?p=7651"},"modified":"2024-01-29T09:32:20","modified_gmt":"2024-01-29T15:32:20","slug":"install-performance-co-pilot-via-ansible-and-ascender","status":"publish","type":"post","link":"https:\/\/gregsowell.com\/?p=7651","title":{"rendered":"Install Performance Co-pilot Via Ansible And Ascender"},"content":{"rendered":"<p><a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/thumbnail-template-1.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/thumbnail-template-1-300x169.jpg\" alt=\"\" width=\"300\" height=\"169\" class=\"aligncenter size-medium wp-image-7665\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/thumbnail-template-1-300x169.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/thumbnail-template-1-768x432.jpg 768w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/thumbnail-template-1-1024x576.jpg 1024w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/thumbnail-template-1.jpg 1280w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><br \/>\nPerformance co-pilot(PCP) is a suite of tools used for performance monitoring for a variety of things.  We see it used quite a bit in the HPC space to either squeeze as much performance out of a system as possible or to troubleshoot performance issues.  It can often be tedious to install and manage\u2026unless, of course, you use automation!<\/p>\n<p>I\u2019ll describe my architecture, review my playbooks, and have a look at it all working.<\/p>\n<h2>Video Demo<\/h2>\n<p><iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/RbfJinLLaJU?si=DCmDexEP8NNke_5T\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/p>\n<h2>How It Works<\/h2>\n<p>PCP has a LOT of components and options, I really intend to just describe how I\u2019m configuring it.<br \/>\n<a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_8.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_8-300x118.jpg\" alt=\"\" width=\"300\" height=\"118\" class=\"aligncenter size-medium wp-image-7653\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_8-300x118.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_8.jpg 695w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><br \/>\nFirst, what is a \u201cCollection host\u201d?  Any regular server or VM running PCP to gather info on itself is considered a collection host.  So most of the configured hosts will be collection hosts.<\/p>\n<p><a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_9.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_9-300x134.jpg\" alt=\"\" width=\"300\" height=\"134\" class=\"aligncenter size-medium wp-image-7654\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_9-300x134.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_9.jpg 706w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><br \/>\nOnce a collector is configured an admin will generally SSH into it to access the PCP data.  These hosts can also run something like redis with grafana to graph info, which means the admin is going straight to the host either way.<\/p>\n<p><a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_10.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_10-300x150.jpg\" alt=\"\" width=\"300\" height=\"150\" class=\"aligncenter size-medium wp-image-7655\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_10-300x150.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_10.jpg 660w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><br \/>\nWhen your environment begins to grow it can be a bit tedious to connect to each host to access PCP info.<\/p>\n<p><a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_11.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_11-300x141.jpg\" alt=\"\" width=\"300\" height=\"141\" class=\"aligncenter size-medium wp-image-7656\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_11-300x141.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_11-768x360.jpg 768w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_11.jpg 892w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><br \/>\nThis is where a \u201cMonitoring Host\u201d comes in.  A monitoring host stores info from multiple collection hosts.  This means an admin only needs to connect to the monitoring host to gain insight about any of the collection hosts\u2026a one-stop-shop as it were.<\/p>\n<p>You can either push or pull data.  If you push data from the collectors they will incur some additional overhead.  If you pull from the monitoring host, it will incur the additional cost, which is less likely to skew your performance metrics from the collection hosts.<\/p>\n<p>I\u2019ve also seen some data saying that a monitoring host should be capped somewhere around a thousand collectors.<\/p>\n<h2>Playbooks<\/h2>\n<p>All of my playbooks can be found here in <a href=\"https:\/\/github.com\/gregsowell\/ansible-performance-copilot\">my git repository<\/a>.<\/p>\n<p><strong>pcp-install.yml<\/strong>. This playbook connects to PCP collectors and configures them to both collect locally and prepares them to allow monitoring hosts to access them:<\/p>\n<pre class=\"gs-code\"><code class=\"language-plaintext\">---\r\n- name: Install\/configure PCP on various hosts\r\n  hosts: pcp-hosts\r\n  gather_facts: false\r\n  vars:\r\n    # Services to be enabled\/started\r\n    enable_services: \r\n      - pmcd\r\n      - pmlogger\r\n    \r\n    # The subnets or ranges of hosts allowed to connect to clients to fetch info\r\n    remote_subnets:\r\n      - 10.0.*\r\n      - 192.168.5.10\r\n\r\n  tasks:\r\n  # dnf install required pcp packages\r\n  - name: Install pcp packages\r\n    ansible.builtin.dnf:\r\n      name: &quot;{{ item }}&quot;\r\n      state: latest\r\n    loop:\r\n      - pcp\r\n      - pcp-system-tools\r\n    notify: restart pcp\r\n\r\n  - name: Configure the pmcd process(add all of the allowed subnets)\r\n    ansible.builtin.blockinfile:\r\n      path: \/etc\/pcp\/pmcd\/pmcd.conf\r\n      block: &quot;{{ lookup(&#039;ansible.builtin.template&#039;, &#039;pmcd-access.j2&#039;) }}&quot;\r\n      insertafter: &quot;\\\\[access\\\\]&quot;\r\n    notify: restart pcp\r\n\r\n  - name: Configure the pmcd options to listen on the correct IP\r\n    ansible.builtin.lineinfile:\r\n      path: \/etc\/pcp\/pmcd\/pmcd.options\r\n      line: &quot;-i {{ hostvars[inventory_hostname].ansible_host }}&quot;\r\n\r\n  - name: Enable pmcd listening ports on firewall\r\n    ansible.posix.firewalld:\r\n      port: 44321\/tcp\r\n      permanent: true\r\n      immediate: true\r\n      state: enabled\r\n    ignore_errors: true\r\n\r\n  - name: Enable selinux for pmcd services\r\n    ansible.builtin.shell: &quot;{{ item }}&quot;\r\n    ignore_errors: true\r\n    loop:\r\n      - setsebool -P pcp_read_generic_logs on\r\n      - setsebool -P pcp_bind_all_unreserved_ports on\r\n\r\n  - name: Start and enable pcp services\r\n    ansible.builtin.service:\r\n      name: &quot;{{ item }}&quot;\r\n      state: started\r\n      enabled: true\r\n    loop: &quot;{{ enable_services }}&quot;\r\n\r\n  handlers:\r\n  - name: restart pcp\r\n    ansible.builtin.service:\r\n      name: &quot;{{ item }}&quot;\r\n      state: restarted\r\n    loop: &quot;{{ enable_services }}&quot;<\/code><\/pre>\n<p>I\u2019m going to point out some things of note in the above playbook.<br \/>\nFirst is the <em>remote_subnets<\/em> variable.  This should be populated with the IP or subnet of your monitoring hosts.  It\u2019s essentially an access list of who is allowed to connect in to retrieve PCP data.<\/p>\n<p>Most of the tasks are pretty straightforward, but I thought I would have a look at one that includes a jinja2 template:<\/p>\n<pre class=\"gs-code\"><code class=\"language-plaintext\">  - name: Configure the pmcd process(add all of the allowed subnets)\r\n    ansible.builtin.blockinfile:\r\n      path: \/etc\/pcp\/pmcd\/pmcd.conf\r\n      block: &quot;{{ lookup(&#039;ansible.builtin.template&#039;, &#039;pmcd-access.j2&#039;) }}&quot;\r\n      insertafter: &quot;\\\\[access\\\\]&quot;\r\n    notify: restart pcp<\/code><\/pre>\n<p>This replaces a block of code using the blockinfile module, but I\u2019m pulling that block from a dynamic j2 template(in the templates folder) named <strong>pmcd-access.j2<\/strong>:<\/p>\n<pre class=\"gs-code\"><code class=\"language-plaintext\">{% for item in remote_subnets %}\r\nallow hosts {{ item }} : fetch;\r\n{% endfor %}<\/code><\/pre>\n<p>Taking a look at the template above you can see I have a simple \u201cfor loop\u201d.  I loop over the contents of <em>remote_subnets <\/em>and fill out the allow hosts section based on it.  Anything inside of <em>{% %}<\/em> is omitted from the actual output of the template.<\/p>\n<p>Now that the PCP collectors are installed and configured, I\u2019ll run the <strong>pcp-monitor.yml<\/strong> playbook to configure the monitor host:<\/p>\n<pre class=\"gs-code\"><code class=\"language-plaintext\">---\r\n- name: Install\/configure PCP monitor host\r\n  hosts: pcp-monitor\r\n  gather_facts: false\r\n  vars:\r\n    # Services to be enabled\/started\r\n    enable_services: \r\n#      - pmcd\r\n      - pmlogger\r\n    \r\n    collection_directory: \/var\/log\/pcp\/pmlogger\/\r\n\r\n    # Do you want to set the pmlogger config files to use host IP address instead of inventory_hostname\r\n    config_via_host: true\r\n\r\n  tasks:\r\n  # - name: debug data\r\n  #   ansible.builtin.debug:\r\n  #     var: hostvars[item]\r\n  #   loop: &quot;{{ groups[&#039;pcp-hosts&#039;] }}&quot;\r\n\r\n  - name: Install pcp packages\r\n    ansible.builtin.dnf:\r\n      name: &quot;{{ item }}&quot;\r\n      state: latest\r\n    loop:\r\n      - pcp\r\n      - pcp-system-tools\r\n    notify: restart pcp\r\n\r\n  - name: Create config file for each pcp-host\r\n    ansible.builtin.template:\r\n      src: pmlogger-monitor.j2\r\n      dest: &quot;\/etc\/pcp\/pmlogger\/control.d\/{{ item }}&quot;\r\n    loop: &quot;{{ groups[&#039;pcp-hosts&#039;] }}&quot;\r\n    notify: restart pcp\r\n    \r\n  - name: Create collector host directories by looping over pcp-hosts group\r\n    ansible.builtin.file:\r\n      path: &quot;{{ collection_directory }}{{ item }}&quot;\r\n      state: directory\r\n      mode: &#039;0777&#039;\r\n    loop: &quot;{{ groups[&#039;pcp-hosts&#039;] }}&quot;\r\n\r\n  - name: Start and enable pcp services\r\n    ansible.builtin.service:\r\n      name: &quot;{{ item }}&quot;\r\n      state: started\r\n      enabled: true\r\n    loop: &quot;{{ enable_services }}&quot;\r\n\r\n  handlers:\r\n  - name: restart pcp\r\n    ansible.builtin.service:\r\n      name: &quot;{{ item }}&quot;\r\n      state: restarted\r\n    loop: &quot;{{ enable_services }}&quot;<\/code><\/pre>\n<p>Again, I\u2019ll try and point out the less obvious or perhaps more interesting parts of the above playbook.  <\/p>\n<p>The variable <em>collection_directory <\/em>is where the collected PCP data from the collectors will be stored.<\/p>\n<p>The <em>config_via_host <\/em>variable is one I put in especially for my lab environment.  When the config files are created, they point to a host to collect.  If this variable is set to true, then the host\u2019s IP address will be used.  If it\u2019s set to false, then the <em>inventory_hostname <\/em>will be used(generally a Fully Qualified Domain Name(FQDN)).<\/p>\n<p>In the previous playbook I used a template, and I\u2019m using one here in the monitor host configuration also in the following task:<\/p>\n<pre class=\"gs-code\"><code class=\"language-plaintext\">  - name: Create config file for each pcp-host\r\n    ansible.builtin.template:\r\n      src: pmlogger-monitor.j2\r\n      dest: &quot;\/etc\/pcp\/pmlogger\/control.d\/{{ item }}&quot;\r\n    loop: &quot;{{ groups[&#039;pcp-hosts&#039;] }}&quot;\r\n    notify: restart pcp<\/code><\/pre>\n<p>Here I\u2019m using the template module directory rather than the template lookup plugin.  Let\u2019s examine the reference <strong>pmlogger-monitor.j2<\/strong> template:<\/p>\n<pre class=\"gs-code\"><code class=\"language-plaintext\">{% if config_via_host %}\r\n{{ hostvars[item].ansible_host }} n n PCP_LOG_DIR\/pmlogger\/{{ item }} -r -T24h10m -c config.{{ item }}\r\n{% else %}\r\n{{ item }} n n PCP_LOG_DIR\/pmlogger\/{{ item }} -r -T24h10m -c config.{{ item }}\r\n{% endif %}<\/code><\/pre>\n<p>This one uses a conditional \u201cif else\u201d statement, rather than just a loop.  This is where I check if the collector host should be referenced via the <em>inventory_hostname <\/em>or via the <em>ansible_host<\/em>.<\/p>\n<h2>Executing\/Troubleshooting Automation<\/h2>\n<p><strong>Configure\/Install\/Troubleshoot Collector<\/strong><br \/>\nOnce you\u2019ve added your inventories, projects, credentials and job templates, you can execute the automaton for installing the collectors:<br \/>\n<a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_1-1.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_1-1-300x234.jpg\" alt=\"\" width=\"300\" height=\"234\" class=\"aligncenter size-medium wp-image-7657\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_1-1-300x234.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_1-1-768x600.jpg 768w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_1-1.jpg 1001w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>If you want to test the collector host, you can pretty easily do it by SSHing in and issuing the \u201cpcp\u201d command:<br \/>\n<a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_2.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_2-300x107.jpg\" alt=\"\" width=\"300\" height=\"107\" class=\"aligncenter size-medium wp-image-7658\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_2-300x107.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_2.jpg 643w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>If the monitor is getting \u201cconnection refused\u201d, be sure the check the listening ports on the collector with \u201c<em>ss -tlp | grep 44321<\/em>\u201d:<br \/>\n<a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_3.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_3-300x80.jpg\" alt=\"\" width=\"300\" height=\"80\" class=\"aligncenter size-medium wp-image-7659\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_3-300x80.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_3.jpg 649w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p><strong>Configure\/Install\/Troubleshoot Monitor<\/strong><br \/>\nOnce you run the monitor playbook you should see the successful message:<br \/>\n<a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_4.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_4-300x175.jpg\" alt=\"\" width=\"300\" height=\"175\" class=\"aligncenter size-medium wp-image-7660\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_4-300x175.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_4-768x448.jpg 768w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_4-1024x598.jpg 1024w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_4.jpg 1069w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Now, if you want to test the monitor host, you can SSH into it and check the collection_directory.  In my case I had it as \u201c<em>\/var\/log\/pcp\/pmlogger\/<\/em>\u201d:<br \/>\n<a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_5.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_5-300x102.jpg\" alt=\"\" width=\"300\" height=\"102\" class=\"aligncenter size-medium wp-image-7661\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_5-300x102.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_5.jpg 542w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>You can see here my PCP collector Greg-rocky9 folder is showing up, but is there data inside?:<br \/>\n<a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_6.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_6-300x210.jpg\" alt=\"\" width=\"300\" height=\"210\" class=\"aligncenter size-medium wp-image-7662\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_6-300x210.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_6.jpg 523w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>This folder is full of data.  If it wasn\u2019t I would do a \u201c<em>tail pmlogger.log<\/em>\u201d in that folder to get an idea of what was happening:<br \/>\n<a href=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_7.jpg\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_7-300x44.jpg\" alt=\"\" width=\"300\" height=\"44\" class=\"aligncenter size-medium wp-image-7663\" srcset=\"https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_7-300x44.jpg 300w, https:\/\/gregsowell.com\/wp-content\/uploads\/2024\/01\/Screenshot_7.jpg 435w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<h2>Conclusion<\/h2>\n<p>While PCP data may not be for everyone, it can, quite easily, be configured.  The trick about performance data is that if you have a performance issue, you can\u2019t go back in time and enable the data collection, so why not go ahead and start collecting BEFORE there\u2019s an issue &#x1f642;.<\/p>\n<p>As always, thanks for reading.  If you have any questions or comments, I\u2019d love to hear them.  If you use PCP in your environment, I\u2019d love to hear about that also!  If we can help you on your automation journey, please reach out to me.<\/p>\n<p>Good luck and happy PCP automating!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Performance co-pilot(PCP) is a suite of tools used for performance monitoring for a variety of things. We see it used quite a bit in\u2026<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[59,14,13],"tags":[],"class_list":["post-7651","post","type-post","status-publish","format-standard","hentry","category-ascender","category-linux","category-server"],"_links":{"self":[{"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/posts\/7651","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/gregsowell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7651"}],"version-history":[{"count":3,"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/posts\/7651\/revisions"}],"predecessor-version":[{"id":7666,"href":"https:\/\/gregsowell.com\/index.php?rest_route=\/wp\/v2\/posts\/7651\/revisions\/7666"}],"wp:attachment":[{"href":"https:\/\/gregsowell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7651"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gregsowell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7651"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gregsowell.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7651"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}