Suppose we want our agent to help with local development, but do not want it running shell commands or editing files without explicit permission. There is a built-in policy we can add to ask the user for approval before allowing the agent to execute any file or shell operations.

You can add the policy to your current session from the Omnigent web app or desktop app by selecting the info button (ⓘ) in the top-right corner, clicking + next to policies, and selecting "Require Approval for File & Shell Operations."

You can also simply ask the agent to add the policy to your session—Omnigent supplies coding agents with the tools they need to add policies themselves.

Now, whenever the agent attempts a file or shell operation, it will first ask the user for permission.

You may want to implement a policy over actions or session states that are not covered by built-in policies. Maybe you want to limit the use of a specific tool, or block an action once sensitive data has entered the session. With Omnigent, you can define custom policies to meet your specific needs.

Fundamentally, a custom policy is a Python function that receives a PolicyEvent and returns a PolicyResponse ( ALLOW, ASK, or DENY).

A PolicyEvent describes something that just happened or is about to happen, such as a message arriving, the agent about to call a tool, a tool returning a result, or the agent about to reply. In our Python code, we can match on event["type"] to pick out the event we care about, and read event["session_state"] to see what has already happened this session. The example below gives the agent a budget of free shell calls, then asks for approval on every one after that.

 from omnigent.policies.schema  import PolicyEvent, PolicyResponse

 FREE_SHELL_CALLS  = 3

 def  budget_shell_calls(event: PolicyEvent)  -> PolicyResponse  |  None:
     if event.get( "type")  !=  "tool_call":
         return  None
     if (event.get( "data")  or {}).get( "name")  !=  "sys_os_shell":
         return  None

     used  = (event.get( "session_state")  or {}).get( "shell_calls_used", 0)
     if used  >= FREE_SHELL_CALLS:
         return {
             "result":  "ASK",
             "reason": f "That's shell call #{used  + 1}  this session — approve it?",
        }

     return {
         "result":  "ALLOW",
         "state_updates": [
            { "key":  "shell_calls_used",  "action":  "increment",  "value": 1}
        ],
    }

This policy is stateful. It reads a counter out of event["session_state"] and returns a state_updates list to bump that counter each time it allows a call. Omnigent persists the counter across the session, so the same echo command is allowed the first three times and paused the fourth—nothing changed but the history. See Contextual Policies in Omnigent for deeper examples, like a running cost total.

You can attach this custom policy directly to your agent config, the YAML file omni run reads:

guardrails:
  policies:
    budget_shell_calls:
      type: function
      on: [tool_call]
      function:
        path: myorg_policies.budget_shell_calls

To use the policy, save the function in a file next to the agent config. Stop the running server if you have one, then start the agent:

omni stop
omni run agent.yaml

Run this from the directory where both files are located. omni run resolves your module against the directory you launch from.

You can also register it on the server, which makes it selectable from the UI for everyone on that server. Add your module to the policy_modules: list in the server config and export a POLICY_REGISTRY list from it; the server scans that list at startup. See the docs for the entry format.

Not every policy needs a Python file. For simple, stateless checks you can write the whole policy as a single Common Expression Language (CEL) expression, right in the agent config. Here's a policy that denies any shell command containing rm -rf:

guardrails:
  policies:
    block_rm_rf:
      type: function
      on: [tool_call]
      function:
        path: omnigent.policies.builtins.cel.cel_policy
        arguments:
          expression: >-
            event.type == "tool_call" && event.target == "sys_os_shell"
              && event.data.arguments.command.contains("rm -rf")
              ? {"result": "DENY", "reason": "Recursive delete is blocked."}
              : {"result": "ALLOW"}

Only sys_os_shell calls get checked: the tool-name test comes first, so calls to other tools—which have no command field to read—pass through without erroring. And unlike the Python policy above, a CEL policy is just an expression with nothing to load, so the agent can add one to a live session immediately, no restart needed.

Policies look at distinct events, and optionally at tracked state, and return ALLOW, ASK, or DENY. Omnigent ships with built-in policies for common needs like safety and cost control, and you can write your own for boundaries specific to your tools, data, and risk tolerance.

Try it yourself: add one to a live session and watch what happens when it's triggered. Next, think about what actions you don't want your agents to take unsupervised, and make it ASK first.

I have been using Claude Code and OpenAI Codex for months now. The underlying models clearly drive much of the effectiveness of these tools—the much commented-on jumps in performance over the past few months are clear demonstrations of the centrality of model performance.

But models frequently leapfrog each other in terms of performance (real or perceived). A new model comes out and tops leaderboards and benchmarks—but many developers tend to stick to, or adopt, a harness that enforces use of what might be considered a "lesser" model (though, at this point, we're splitting hairs. The top models are all incredibly capable).

Right now, Claude Opus 4.6, GPT Codex 5.3, and Gemini 3.1 Pro are vying for the top positions on the terminal bench 2 benchmark, with some variations depending on the harnesses in which they are running.

Even though Codex 5.3 currently hold the top slot, the Claude Code coding agent harness is much more popular than the OpenAI harness. Compare their npm downloads, for example—an incomplete, but directionally correct datapoint.

All of this is to say: the harness is crucially important. The harness is more responsible for the user's experience of working with a given coding agent than the underlying intelligence is. It's also much easier to modify and customize than the underlying model. More on that later. First: what is an agent harness?

In most of the articles I've read, the term "harness" is used but not defined. I use it very broadly. The harness is the loop in which the LLM is invoked and in which it interacts with user(s) and/or its environment.

A single API call to an LLM does not involve a harness, in this formulation. But even a simple chat loop constitutes a harness. Other features of a harness might include:

• tool calling (bash execution, web search, web fetch, etc.)
• context management (compaction, clearing, summarization, checkpointing, knowledge/discovery of tools, skills, etc.)
• instrumentation/observability (capturing traces, running automated evaluations)
• subprocess management: for when an agent launches long-running processes like web servers or subagents

You probably aren't in a position to build your own Claude Opus model. But you can build your own harness. If nothing else, it will help you to understand what is possible and how much control you have over your agentic coding environment. Claude Code is great. I'm almost certainly going to keep using it. But having your own agent harness gives a great platform for experimenting with different patterns of context handling, tool use—all the stuff that isn't the underlying intelligence.

In the rest of this post, we'll make a simple extensible harness. This will be the foundation for some future experiments I have planned. More on that later.

My goal here was to build a very simple but extensible agent harness. I started by sketching out what this means:

1. Simple control flow: it's just a loop.
2. Everything (human input, tool calls, tool responses, agent responses) is an explicit, understandable event
3. Simple defaults. I'm not doing anything clever at this phase.
4. Provider agnostic—don't anchor to a specific model/provider.
5. Observability is built in but non-blocking

The goals, at this phase, are interpretability and transparency. Optimization and experimentation will come later.

This harness is enough of a foundation to start running some experiments on. We have the core loop. It's modular and customizable. The results are all traced in MLflow.

You might feel like this post was a little thin. And you'd be right to feel that way. That's the point. Working with the major coding agents today can feel like petitioning a genie for help and hoping that its mystical machinations return what you need. What I hope I communicated is that agent harnesses can be very simple.

From here, we'll be doing some more experiments with features we can layer into our agent harness. I suspect, for example, that there are far better ways to handle context; that chat transcripts are extremely poor in terms of useful information density when working toward specific outcomes. I'll write about that, and some early experiments, next time.

Suppose I have the following window configurations open. When I say "window configuration," I mean "a set of visible windows in a particular arrangement."

1. My init.el file along with a docs page on desktop-save-mode.

   

2. The org file for the blog I am working on now, a scratch buffer, and my blog's build script.

   

Naively navigating from (1) to (2) requires changing the leftmost buffer to my blog draft, splitting the buffer on the right, and switching the two windows on the right to the scratch and build script buffers. Switching back requires deleting on of the windows on the right and navigating back to the original two buffers.

This is a pain.

I previously wrote about navigating between points with registers. You can register a point with and then return to it with .

You can also register window configurations. The process is essentially the same.

1. Call window-configuration-to-register with .

2. Select a register. Any single character (including non-alphanumeric characters such as emojis) can be used to set a register. So e.g. will save the window configuration to register a. will save it to register 1. You can read more about registers here.

   

3. Navigate away from your saved window configuration. When you want to get back to it, you can do so with and then select the register you want to return to from the *Register Preview* menu.

   

   This will immediately return you to the window configuration you saved above.

Once we register a couple of different window configurations, it becomes very easy to swap between them!

Registers do not persist across emacs sessions—when you restart emacs, your registers are gone.

One approach to persisting the state of emacs across sessions is with desktop-save-mode, which saves a .emacs.desktop file to your user-emacs-directory (or whatever other location you specify), and then reads from this file when emacs is restarted or when you manually call desktop-read.

In my limited testing, this approach is imperfect. Some buffers are not restored, and window registers seem to be saved in a corrupted and unusable form (even though register-alist is part of the desktop-globals-to-save list, which specifies what is preserved across sessions).

Still, it's better than nothing? I'm sure there are other solutions out there.

For me, this is not much of a problem. I don't often exit emacs.

It's simple:

1. and then select the register (any single character)
2. Navigate away from your present window configuration

3. , choose the register to which you saved your configuration in (1)

   And you're back where you started!

MLflow tracing is a powerful AI observability tool that enables you to capture all of the inputs, outputs, and metadata associated with every step of an AI model or Agent execution chain. It works with many different providers, such as OpenAI, Anthropic, LangChain, and LlamaIndex via a single line of code. Tracing provides granular insight into the entire execution chain of AI applications, including tool calls, retrievals, AI responses, any anything else you might want to include.

But capturing the traces is just the first step: once we have all of the trace data, we need to use it to make our AI applications better. In addition to its existing sophisticated evaluation functionality, which enables us to add human, AI, and programmatic assessments to traces, MLflow 3.4 introduced an MLflow MCP server to give AI applications like Claude Desktop and code assistants like Cursor the ability to interact with traces.

The MCP server lets these tools search and analyze trace data, log feedback, manage metadata, and delete traces and assessments.

In the remainder of this post, we will see how to configure and use the MCP server.

Now that we have set up MLflow and generated some sample traces, let's explore them with the help of AI models! We will show how to do this with with Claude Desktop and Claude Code.

There's something satisfying about using AI to debug AI. The MLflow MCP server closes the loop between capturing traces and actually using them: your AI assistant can now help you understand why your other AI assistant failed.

The MLflow docs suggest some use cases, but the real value comes from exploring your own patterns:

• Ask Claude to compare successful vs. failed traces to identify common failure modes
• Have it search for traces with specific token usage patterns when you're trying to optimize costs
• Use it to find traces where retrieval returned irrelevant content, then iterate on your chunking strategy
• Let it spot when certain model configurations consistently produce better results
• Give Claude Code access to both your agent code and its traces so it can review failures, suggest fixes, and help you iterate without leaving your editor

The setup takes five minutes, but once configured, your trace data becomes something you can have a conversation with instead of handcrafting search queries or digging through the UI.

In the first post in this series, we learned (1) what an MCP server is, (2) how to define a simple MCP server with a single tool, and (3) how to install that server to Claude Desktop. Let's briefly recap these points.

1. MCP servers provide a standardized interface to resources (such as databases, file systems, tools, etc.) that LLMs can access. An MCP client corresponding to that server facilitates requests to the server, receives responses, and relays them to the host or user. MCP clients and servers are usually used in the context of a host application such as an IDE or chat app; the host is what the end user typically interacts with.

2. The following code snippet defines an MCP server that exposes a single tool.

    #  server.py
   
    #  install the mcp sdk with pip install "mcp[cli]"
    from mcp.server.fastmcp  import FastMCP
   
    mcp  = FastMCP( "SecretServer")
   
    @mcp.tool()
    def  get_secret_number()  ->  int:
        """Returns a predefined secret number."""
        return 13
   

   The tool, get_secret_number, just returns a "secret number." If you ask an LLM with access to this MCP server for the secret number, it should invoke this function in order to retrieve it.

3. We installed the server to Claude Desktop with mcp install server.py.

This was an example of defining and using a tool with MCP. In this post, we will learn about other MCP primitives allowing the MCP server to expose data and enable specific workflows.

Resources provide a way to give your LLMs access to data. The data can include database records, file contents, API responses, images, and more. Getting data from a resource in an MCP server is like making an HTTP GET request. The expectation is that it will return data without any side effects.

In fact, the secret number method we defined previously would be a perfect candidate for a resource. It returns data and does nothing else. Let's show how to define an equivalent resource, this time to return a secret letter.

We can expand our MCP server with this new resource as follows:

 #  server.py

 #  install the mcp sdk with pip install "mcp[cli]"
 from mcp.server.fastmcp  import FastMCP

 mcp  = FastMCP( "SecretServer")

 @mcp.tool()
 def  get_secret_number()  ->  int:
     """Returns a predefined secret number."""
     return 13

 @mcp.resource( "data://secret_letter")
 def  get_secret_letter()  ->  str:
     """Returns a predefined secret letter."""
     return  "L"

The @mcp.resource decorator takes the resource's unique URI as an argument. Clients use this URI to request data.

MCP prompts let you set up parameterized prompt templates. Just like with tools and resources, we use the @mcp.prompt() decorator to define prompts with the Python SDK. Let's update our server with a simple prompt that can generate a haiku on a topic.

 #  server.py

 from mcp.server.fastmcp  import FastMCP, Context

 mcp  = FastMCP( "SecretServer")

 @mcp.tool()
 def  get_secret_number()  ->  int:
     """Returns a predefined secret number."""
     return 13

 @mcp.resource( "data://secret_letter")
 def  get_secret_letter()  ->  str:
     """Returns a predefined secret letter."""
     return  "L"

 @mcp.tool()
 async def  retrieve_secret_letter_tool(ctx: Context)  ->  str:
     """Tool to retrieve the secret letter from the corresponding resource"""
     resources  =  await ctx.read_resource( "data://secret_letter")
     return resources[0].content

 @mcp.prompt()
 def  haiku(topic:  str)  ->  str:
     """Generates a user message asking for an explanation of a topic."""
     return f "Can you please write a haiku about '{topic} '?"

In this two-part series, we have seen how to set up an MCP server for use with Claude Desktop and how to define tools, resources, and prompts on that server. This should have given you what you need to take your first steps into the world of MCP servers.

Using the MCP Python SDK makes it straightforward to define these MCP server components. As we saw above, they all follow the same general pattern: we define a function, which may or may not take arguments, and decorate it with the decorator corresponding to the component type.

While this introduction only scratches the surface, MCP's real power comes when you start connecting your LLMs to actual systems like databases, APIs, and file systems. The standardized interface means you can combine MCP servers from different sources without worrying about compatibility issues.

Anthropic's Model Context Protocol (MCP) is intended to standardize the way applications give context (data, tools, prompts, etc.) to LLMs.

We can think of an application using MCP as having three components:

1. Host: the host is the application, such as an IDE or a chat app, that manages connections to one or more MCP servers. The host application is what the end user typically interacts with.
2. MCP Server(s): MCP servers provide a standardized interface to resources (such as databases, file systems, tools, etc.) that LLMs can access.
3. Client(s): clients are components of the host application that maintain the connection with the MCP server(s). Each client corresponds to one server. A client facilitates requests to the server, receives responses, and relays them to the host or user.

For example, a user might type the following into a chat app (the host): "Give me information about order ORD-1234." The client component of the application will pass this query, along with a list of available tools, to the LLM. Suppose one of the tools gives the ability to query a database of order statuses. The LLM will respond with a "tool call" specifying the tool to use (the database query tool) and the arguments to provide (the order id). The client will then invoke the tool from the MCP server, get the results of the database query, and pass them along to the next step, which might mean asking the LLM to respond to the user based on the results.

To get started, we will set up an MCP server that exposes a single function, get_secret_number, that simply returns a "secret number" to the LLM. When you ask an LLM with this capability for the secret number, it will invoke the tool and return the "secret number," verifying that it successfully used the MCP server.

We will install this server to the Claude desktop app, allowing you to use the server without setting up a custom client.

To recap, in this post, we set up a very minimal MCP server using the FastMCP method from the Python MCP SDK package. We defined a single tool that returns a "secret number" when invoked. We then installed the server to the Claude desktop app and confirmed that Claude could successfully invoke the tool via the MCP server to retrieve the secret number.

This minimal implementation is a good starting point for further exploration with MCP. From here, you can add more components to the server and test them with the Claude Desktop app. Some suggestions for next steps:

• Learn about resources, a mechanism for MCP servers to expose data to clients/LLMs. You can either directly expose resources or you can add tools enabling the LLM to query resources. More on this in a future post!
• Define and use prompts in your MCP server.

I will discuss these additional MCP components in my next post on the topic, and then write about how to create a custom MCP Client in Python.

For a while, whenever I used eglot (mostly for Python projects) and then shut it down with e.g. M-x eglot-shutdown, all subsequent attempts to save any files in Emacs would return the following error: (jsonrpc-error-message . "No current JSON-RPC connection"). The save would generally succeed, so this was more annoying than anything. But it was annoying.

Some quick searching online did not yield any useful discussions of how to fix the issue. And neither Claude nor Perplexity had much to offer, either. So I decided to use this as an excuse to learn more about built-in Emacs introspection and debugging tools.

I went in with the following information:

1. The issue was related to eglot. Even though the error message did not reference eglot, the issue consistently appeared after using eglot and then shutting it down.
2. The issue was triggered when I saved files.

Given this pattern, I started by looking for hooks related to saving.

I wasn't entirely sure what I was looking for. But I knew from previous reading that hooks are variables that, by convention, include the word hook in their names. This already provides plenty to work with.

The apropos-variable function takes a pattern or a list of words and returns an *Apropos* buffer with a list of matching variables. In this case, I interactively used apropos-variable with M-x apropos-variable RET save hook to search for variables matching save and hook. This returned a couple of good candidates for further exploration!

The after-save-hook and before-save-hook variables look especially promising. We can inspect those further for anything eglot-related by selecting them from the *Apropos* buffer or searching for them with C-h v . Following this approach showed me that:

1. The value of before-save-hook is nil.
2. The value of after-save-hook is (eglot-format).

This is already very useful and points toward some directions for fixing the issue! after-save-hook is a "Normal hook that is run after a buffer is saved to its file." This would explain why (1) the issue is associated with saving files and (2) the saves are successful in spite of the error (the hook is run after the save, not before).

I wanted to confirm that this hook was causing the issue so I used M-x toggle-debug-on-error to get a more detailed backtrace. In reality, this is where I should have started—the error trace provides much more specific and useful information than the short error message returned in the echo area. When I tried to save a file with debugging enabled, I received the following in a Backtrace buffer.

Debugger entered--Lisp error: (jsonrpc-error "No current JSON-RPC connection" (jsonrpc-error-code . -32603) (jsonrpc-error-message . "No current JSON-RPC connection"))
  jsonrpc-error("No current JSON-RPC connection")
  eglot--current-server-or-lose()
  eglot-server-capable(:documentFormattingProvider)
  eglot-server-capable-or-lose(:documentFormattingProvider)
  eglot-format()
  run-hooks(after-save-hook)
  #(t)
  polymode-with-current-base-buffer(# t)
  apply(polymode-with-current-base-buffer # t)
  basic-save-buffer(t)
  save-buffer(1)
  funcall-interactively(save-buffer 1)
  command-execute(save-buffer)

You can read this backtrace from bottom to top. After saving the buffer, we see that run-hooks(after-save-hook) runs, which results in eglot-format() being run. The final function called before the error is eglot--current-server-or-lose(). Inspecting this with C-h f RET eglot--current-server-or-lose tells us that this function returns the "current logical Eglot server connection or error." If I'm saving some random file that is not going to use a Python LSP server, we would expect this to return an error.

Now that we have some understanding of what is happening, how do we fix it?

My initial fix for this issue was to write a simple cleanup script to remove the offending hook function from the after-save-hook.

( use-package eglot
   :straight
  ( :type built-in)
   :hook ((python-mode . eglot-ensure))
   :config
  ( setq eglot-autoshutdown t)
  
  ( defun  my-eglot-shutdown-cleanup ( &rest _)
     "Perform thorough cleanup after Eglot shutdown."
    (remove-hook 'after-save-hook #'eglot-format nil)
  (advice-add 'eglot-shutdown  :after #'my-eglot-shutdown-cleanup))

This isn't perfect. It does prevent saving from returning an error after I've shut down eglot, resolving a significant nuisance. However, the issue remains when eglot is running and I try to save a buffer without an associated LSP server; i.e., if I am using eglot in a Python buffer but then try to save an org buffer. The eglot-format hook function is still active; there is no running language server to provide formatting for org buffers, so the hook function returns an error.

At this point, it was not yet clear to me how the hook was set in the first place. Deactivating the hook when eglot is not running resolves about 80% of the frustration for me. But I would like to fully resolve the issue. I don't actually want the format-on-save behavior in the first place. It has to be set somewhere. In the next section, I will briefly sketch out my process for identifying the issue.

My first thought was that, perhaps, the hook was being set when eglot was invoked. To check this, I:

1. Called C-h f eglot to find the documentation for the eglot command
2. Followed the link in the help buffer to eglot.el, the source file where the eglot command and related functions are defined.
3. Used consult-line (or, equivalently, isearch or one of the many other tools available for searching buffer text) for the term hook.

This showed me that eglot-format was not, in fact, being set as an after-save-hook function by eglot itself.

So…did I do this myself, somewhere in my config?

I next navigated to my config folder, ~/coffeemacs/, and invoked lgrep to search my various *.el config files for anything related to eglot.

And it turns out, I set this hook myself!

Once I deleted the (add-hook 'after-save-hook ...) call from my config, the issue was fully resolved.

The approaches I used here are nowhere close to comprehensive. Emacs has countless introspection tools and a seemingly-inexhaustible collection of functions and variables that enable you to inspect everything going on in your Emacs setup. Furthermore, it provides a range of ways to search these variables and functions.

The following tools will go a long way toward helping you debug an error in Emacs:

1. Enable debugging on error with M-x toggle-debug-on-error. This will provide a backtrace that will show the source of the error.
2. Search for relevant functions and variables with apropos-function and apropos-variable. You can pass in a list of relevant terms to search for.
3. Get documentation for specific functions and variables with the describe-function ( C-h f) and describe-variable ( C-h v) commands.

Even these relatively simple tools are often enough to identify the source of an issue and do something about it.

Lastly—we're reaching a point where you don't have to do this yourself. You can configure the gptel package with a set of tools—Emacs functions—that will enable it to recursively search for information in docs, manuals, source code, etc. This video provides a good overview of how to get started.

I've mostly thought of the org agenda as, well, an agenda. It's for managing todos, dates, deadlines. It's a very versatile tool for that purpose, and can pull those elements out of a bunch of different org files. This is great.

But it's a lot more general than that. It's not just about building an agenda. We can pull all kinds of structured data out of org files with no need to conceptualize the end result as having anything to do with an agenda.

The problem I set out to solve is simple:

• I keep daily journal files
• I seldom refer back to those files
• Sometimes I write something in those journal files that I want to revisit in the future.

You can see the problem. I'm writing something I intend to revisit in the future in a file I tend not to revisit.

With the org agenda, I can remedy this as follows:

• Apply a tag (or category, or todo status, or another searchable marker) to headers in my journal I want to see in the future
• Build a custom org agenda view that searches only the journal files and displays only those tagged headers.

I did this with org-agenda-custom-commands. It looks like this in my config:

( setq '(( "j"  "Journals" ((tags  "persist")
                         (todo))
         ((org-agenda-files
           (file-expand-wildcards  "~/org/denotes/journal/*.org"))
          (org-agenda-prefix-format  "%-2c:::")))))

This creates a custom view called "journals" that I can select with the "j" command from the agenda menu (from ). Any headline tagged with "persist" will appear in that agenda view. It was also updated to include all TODO entries in the journals, ensuring that I properly resolve or migrate them.

This is a very simple agenda view. There are a few improvements I intend to make in the near future (I will update this post accordingly).

Suppose I have an org mode heading representing a scheduled event, like a class:

 * Class
 SCHEDULED:  <2024-03-26 Tue>

Now suppose this class is scheduled weekly for the next five weeks. We don't necessarily want to manually copy the heading and manually update each timestamp (though, admittedly, it would only take a minute or two). This is where org-clone-subtree-with-time-shift comes in.

When we invoke this function with the cursor on the first heading, we will be prompted for (1) the number of clones to create and (2) the time shift we want to use. In this example, we specify 4 clones (for a total of 5 headings) and a +1w time shift, to shift each heading by one week.

 * Class
 SCHEDULED:  <2024-03-26 Tue>
 * Class
 SCHEDULED:  <2024-04-02 Tue>
 * Class
 SCHEDULED:  <2024-04-09 Tue>
 * Class
 SCHEDULED:  <2024-04-16 Tue>
 * Class
 SCHEDULED:  <2024-04-23 Tue>

In some cases, you may want to change the names of the headers somewhat (such as by adding a counter; class 1, class 2, etc.). There are many different approaches to handle this programmatically, but it's beyond the scope of this short post. In this case, it takes just a few seconds to add a class number to each heading manually.

Most helpfully, here is the documenation from C-h f org-clone-subtree-with-time-shift:

org-clone-subtree-with-time-shift is an interactive Lisp closure in ‘org.el’.

It is bound to C-c C-x c.

(org-clone-subtree-with-time-shift N &optional SHIFT)

Clone the task (subtree) at point N times. The clones will be inserted as siblings.

In interactive use, the user will be prompted for the number of clones to be produced. If the entry has a timestamp, the user will also be prompted for a time shift, which may be a repeater as used in time stamps, for example ‘+3d’. To disable this, you can call the function with a universal prefix argument.

When a valid repeater is given and the entry contains any time stamps, the clones will become a sequence in time, with time stamps in the subtree shifted for each clone produced. If SHIFT is nil or the empty string, time stamps will be left alone. The ID property of the original subtree is removed.

In each clone, all the CLOCK entries will be removed. This prevents Org from considering that the clocked times overlap.

If the original subtree did contain time stamps with a repeater, the following will happen:

• the repeater will be removed in each clone
• an additional clone will be produced, with the current, unshifted date(s) in the entry.
• the original entry will be placed after all the clones, with repeater intact.
• the start days in the repeater in the original entry will be shifted to past the last clone.

In this way you can spell out a number of instances of a repeating task, and still retain the repeater to cover future instances of the task.

As described above, N+1 clones are produced when the original subtree has a repeater. Setting N to 0, then, can be used to remove the repeater from a subtree and create a shifted clone with the original repeater.

For more, you can also consult:

• Entry in the Structure Editing docs
• Mastodon discussion on the topic, including some discussion of workarounds for the aforementioned limitations.

A hook is a list of functions that are run on specifically-defined occasions, triggered by calls to a run-hooks or run-mode-hooks function.

Hooks are very commonly run when major modes (and, often, minor modes) are initialized in an emacs session. They provide a way to set up mode-specific configurations when a mode is initialized or when some specific action is undertaken by the user.

For a specific example, org-tree-slide-mode defines the hooks org-tree-slide-play-hook and org-tree-slide-stop-hook, which define what happens when a slide show is started or stopped. I wanted to increase the font size, decrease the width of the displayed text, and remove the header line when the slide show was active, and reset them when it was finished.

First, a quick glossary of terms:

• a hook is a variable defining a list of functions. These variables typically end in -hook, such as org-mode-hook. Hooks may also end in -functions; these so-called "abnormal hooks" are not the focus of this post. Still, if you're searching for hooks, it is good to know that this pattern exists.
• a hook function is one of the functions that comprise the hook.
• There are normal and abnormal hooks. Most hooks are normal. For the purposes of this post, I am focusing on normal hooks. In a normal hook, none of the hook functions take any arguments, and emacs calls each hook function in the list sequentially. Abnormal hooks may take arguments or exhibit other behaviors when called that require special attention/documentation.

Hooks are run when specific functions call them. Such functions are often called at specific points in the initialization or use of emacs modes, but they are not limited to those circumstances. Here is a minimal example.

Hooks are most commonly encountered in the context of modes. Modes generally define hooks to which users can add functions that will be called at the end of the mode's initialization. For example, I wanted to display line numbers whenever I was in a buffer with code, so I have the following line in my config:

(add-hook 'prog-mode-hook 'display-line-numbers-mode)

Inspecting the hook shows:

prog-mode-hook is a variable defined in ‘prog-mode.el’.

Its value is (outline-minor-mode display-line-numbers-mode) Original value was nil

Normal hook run when entering programming modes.

This variable may be risky if used as a file-local variable. You can customize this variable. Probably introduced at or before Emacs version 24.1.

and, indeed, when I open an e.g. Python buffer, line numbers appear as desired.

I use use-package for managing my emacs packages. use-package declarations allow users to pass a :hooks option in the package declaration in order to add functions to hooks. Hooks can be configured in use-package by defining a cons cell as follows.

( use-package package-name
   :hook
  ('mode-name . 'function-to-add-to-hook)
  )

Note that we do not refer to mode-name-hook in the hook configuration. use-package adds the -hook automatically by default. The above will add function-to-add-to-hook to mode-name-hook.

• My Emacs config. This has not, admittedly, been structured for broad consumption, but with a little searching you can find how I've configured some hooks. In particular, here is my org-tree-slide configuration, which I mentioned at the beginning.
• Emacs Docs, which go into some more detail on abnormal hooks among other topics.

First of all, if you haven't already tried chatgpt-shell, it's worth taking a few minutes to set up and try out. If you live in emacs, you'll probably prefer chatgpt-shell to the chatgpt web interface.

You can find the installation instructions here. I use straight.el for package management, and here is my configuration:

( use-package shell-maker
   :straight ( :host github  :repo  "xenodium/chatgpt-shell"  :files ( "shell-maker.el")
                    :branch  "main"))

( use-package chatgpt-shell
   :after (shell-maker)
   :straight ( :host github  :repo  "xenodium/chatgpt-shell"  :files ( "chatgpt-shell.el")
                    :branch  "main")
   :config
  ( setq chatgpt-shell-openai-key
      ( lambda ()
        (auth-source-pick-first-password  :host  "api.openai.com"))))

( use-package ob-chatgpt-shell
   :straight ( :host github  :repo  "xenodium/chatgpt-shell"  :files ( "ob-chatgpt-shell.el")
                    :branch  "main")
   :ensure t )

chatgpt-shell is built on shell-maker, which is "a way to create shells for any service (local or cloud)." ob-chatgpt-shell provides a way to interact with chatgpt-shell via org babel source blocks. For more details on these, read the documentation (and consider sponsoring Xenodium).

YASnippet is a template system for emacs. It is very powerful, and I'm only familiar with a small fraction of its capabilities. But when I work with ChatGPT in other systems, I almost always use some form of templating system. What does that mean? Suppose I'm working on some kind of text summarization system. On the one hand, I could write a bunch of separate prompts of the form:

Summarize the following text:

Summarize the following text:

Summarize the following text:

But the instruction, "Summarize the following text," is the same each time. So I'd rather put that in a template, which I can fill in as many times as needed with the part of the prompt that actually changes: the prompt to summarize.

This might look like:

( defun  summary_template (text_to_summarize)
  ( interactive  "sText to Summarize: ")
  (concat  "Summarize the following text:\n" text_to_summarize))

(print (summary_template  "Some text"))
(print (summary_template  "Some more text"))
(print (summary_template  "Even more text"))

"Summarize the following text:
Some text"

"Summarize the following text:
Some more text"

"Summarize the following text:
Even more text"

YASnippet provides an easy way for us to define and populate templates for use in chatgpt-shell buffers.

There's a lot more to explore here. YASnippet is a very powerful templating system, and I've only scratched the surface of the ways to use it for developing useful prompt templates for chatgpt-shell (and I haven't tried any for the org babel integration yet). Read the YASnippet manual, write some new snippets, and let me know what you come up with!

This article introduces our work on AI+databases. Our original goal was simple: make a great natural language to SQL translation system. This article talks about some of the challenges and opportunities in that area and introduces a Streamlit app we put together for testing text-to-SQL translation prompts.

I wrote this article after the ChatGPT API was released. We started by using the Codex code completion models but the much lower price of gpt-3.5-turbo made a compelling case for switching to ChatGPT. This proved to be a timely article: shortly after we published this article, OpenAI announced that they were discontinuing the Codex model. This article provides a useful guide for using ChatGPT to get much of the same functionality.

This article presents a simple suite of text-to-SQL translations and compares the performance of the Codex and ChatGPT models.

The Unix Philosophy considers text streams a universal interface, and emphasizes the composability of processes that act on text streams. This is what LLMs do: they act on and return text streams.

This article urges readers to think of how LLMs might interface with other tools. As shown by projects such as LangChain and AutoGPT, their usefulness is multiplied when they are integrated with other projects and tools, not just used for chatting.

In this article, I briefly show how to use the pgvector Postgres extension for semantic search. I exported all of bit.io's docs to a Postgres database, created vector embeddings from the docs, and showed how to use pgvector to query the embeddings. Then I used ChatGPT to generate summaries from the results of semantic search applied to the docs.

In my most recent article, I described how we used a two-step process to generate API calls using ChatGPT. We first used ChatGPT to identify the task that the user wanted to complete, and then we combined the user prompt with a task-specific prompt to generate API call completions. The task-specific prompt left much less room for error and resulted in better completion quality.

A struct, also referred to as a composite type, is "a collection of named fields, an instance of which can be treated as a single value." By default, structs are immutable: they can't be modified after construction. This doesn't work for our use case because want to keep adding messages as the chat continues. So we'll use a mutable struct.

One obvious question: what should the named fields of the struct be? Should the struct define all of the behavior of the model (e.g. model choice, parameters such as temperature and max_tokens, etc.)? Or should it narrowly contain the message comprising the chat?

I think the latter approach makes the most sense. It's possible to change model parameters, and even the model itself, mid-chat. They are features of what we are doing to the chat, not of the chat itself.

With that in mind, here is a possible approach to a struct for ChatGPT.

 """
    struct Chat

Represents a conversation between a user and a chatbot powered by OpenAI's GPT.

# Fields

- `messages::Array{Dict{String, String}}`: An array of dictionaries representing the chat messages.

# Constructors

- `Chat(system_message::String="You are a helpful assistant")`: Create a new `Chat` object with a single system message.

# Example

```julia
chat = Chat("You are a helpful assistant.")
```
This creates a new Chat object with a single message representing the system message "You are a helpful assistant.".
"""
 mutable struct  Chat
    messages:: Array{Dict{String, String}}
    
     function  Chat(system_message= nothing)
         if isnothing(system_message)
            system_message =  "You are a helpful assistant"
         end
        messages = [Dict( "role" =>  "system",  "content" => system_message)]
        new(messages)
     end
 end

This struct includes an inner constructor. Inner Constructor Methods allow for the construction of self-referential objects. In this case, we want to be able to Initialize an instance of Chat with just the system message: we don't want to require the user to provide the whole messages array. That's where the self-referential part comes in. The inner constructor method takes an argument, system_message, nests it in a properly-formatted array of dictionaries, and, using the new function, creates a new instance of the Chat struct with the messages array constructed from the system_message.

We can now make a new chat instance, initialized with a system message, with:

julia_helper = Chat( "You are a helpful assistant who knows a lot about writing Julia code")

Chat([Dict("role" => "system", "content" => "You are a helpful assistant who knows a lot about writing Julia code")])

Now that we have a method for keeping track of the chat history, we need to be able to act on it. For that, we'll define a function.

The purpose of this function is to:

1. Get a prompt from the user
2. Append that prompt to a Chat instance's messages array
3. Query the ChatGPT API with the messages array, possibly with some parameters specifying e.g. the specific model to use, temperature, etc.
4. Append the API response message to the Chat instance's messages array
5. Return the API response.

This function acts on the Chat type. It modifies an instance of Chat in place. Here's the function:

 """
    chat!(chat, message::String, api_key=ENV["OPENAI_API_KEY"]; kwargs...)

Add a new message to the chat history and get a response from the OpenAI GPT-3 API.

# Arguments

- `chat`: A `Chat` object representing the chat history.
- `message`: A string representing the user's message.
- `api_key::String=ENV["OPENAI_API_KEY"]`: Your OpenAI API key. If not provided, the function will attempt to get it from the `OPENAI_API_KEY` environment variable.
- `kwargs...`: Any additional keyword arguments to pass as part of the API request body.

# Returns

A string representing the response from the chatbot.

# Example

```julia
chat = Chat("You are a helpful assistant")
response = chat!(chat, "How are you?")
```

This adds a new message to the Chat object chat, representing the user's message "How are you?", and gets a response from the OpenAI ChatGPT API. The response from the chatbot is returned as a string in the response variable.
"""
 function  chat!(chat:: Chat, message:: String, api_key=ENV[ "OPENAI_API_KEY"]; kwargs...)
     if isnothing(api_key)
        error( "API key is required")
     end
    headers = HTTP.Headers([
         "Authorization" =>  "Bearer $api_key",
         "Content-Type" =>  "application/json",
    ])

    formatted_query = Dict( "role" =>  "user",  "content" => message)

    messages = push!(chat.messages, formatted_query)

     #  Merge the default and keyword parameters
    params = merge(Dict( "model" =>  "gpt-3.5-turbo",  "messages" => messages), kwargs)

     #  Convert the parameters to JSON
    body = json(params)

     #  Make a POST request to the OpenAI API endpoint with the query as data
    response = HTTP.post(
         "https://api.openai.com/v1/chat/completions",
        headers,
        body;
        verbose =  false,
    )

     #  Parse the response body as JSON
    result = JSON.parse(String(response.body))

     #  Append the response to chat.messages
    push!(chat.messages, result[ "choices"][1][ "message"])

     #  Return the text field of the result as a string
     return result[ "choices"][1][ "message"][ "content"]
 end

A quick note about the function name: According to the Julia style guide, we append ! to the names of functions that modify their arguments. Furthermore, inputs that are mutated go before inputs that are not mutated in a function's argument list. The chat function follows both of these conventions.

So, does it work? Let's try it out.

chat!(julia_helper,  "What are the main differences between a Julia Struct and a Python Class?")

""Both Julia `struct` and Python `class` are used for creating custom data types, but there are some differences between them:\n\n1. **Type stability:** One of the most significant differences is that Julia `structs` have a static and immutable type, which makes them more type-stable than Python `classes`. In contrast, Python classes are more dynamic, meaning that their attributes can be modified at runtime.\n\n2. **Performance:** In general, Julia `structs` have better performance than Python `classes` due to its type-stability, just-in-time (JIT) compilation, and parallel processing.\n\n3. **Syntax:** The syntax for defining a Julia `struct` is `struct Name{T<:AbstractType} a::T b::Int end`, while in Python, you define a `class` with `class MyClass: def __init__(self, a, b): self.a = a self.b = b`. \n\n4. **Inheritance:** Both Julia and Python support inheritance, but they have different syntax and behavior. In Julia, you use the keyword ` <: ` to specify that a `struct` is a subtype of another `struct`. In Python, you use parentheses after the class name to indicate which class to inherit from.\n\n5. **Typing:** Julia uses type annotations to specify the type of variables, while Python follows the duck typing philosophy, which means that the type of a variable is determined at runtime based on its behavior.\n\nIn summary, while both Julia `structs` and Python `classes` are flexible and powerful tools for creating custom data types, the main differences lie in their type stability, performance, syntax, inheritance, and typing.""

And does it "remember" earlier parts of the conversation correctly?

chat!(julia_helper,  "I only have the attention span for Twitter. Summarize in 280 characters.")

"Julia structs & Python classes are used for custom data types but differ in: \n1. Type stability: Julia is static, immutable; Python is dynamic.\n2. Performance: Julia > Python due to type-stability, JIT compilation & parallel processing.\n3. Syntax: structs use \"struct Name{T} a::T end;\" & classes use \"class MyClass: def __init__(self):\".\n4. Inheritance: Julia uses \"<:\" to specify subtypes; Python uses parentheses for inheritance.\n5. Typing: Julia uses type annotation; Python uses duck-typing."

Well, it's a little longer than I asked for. But clearly we successfully sent the message history in the second API request.

There are a few additional avenues I want to explore, in no particular order:

1. What happens if we counterfeit a message history? That is, what if we send a message history with fake "assistant" messages? Will the assistant mimic the fake responses?
2. Can we make a Julia REPL mode that gives rapid access to a ChatGPT assistant?
3. Can we make a (private) replacement for ChatGPT Plus using the ChatGPT API? It would likely be considerably cheaper. And doing it in Julia would be an interesting project.
4. Can we use the streaming output in Julia? How does that work?

Here is the API documentation on OpenAI's website. An example request with curl looks like this:

curl https://api.openai.com/v1/chat/completions  \
  -H  'Content-Type: application/json'  \
  -H  'Authorization: Bearer YOUR_API_KEY'  \
  -d  '{
  "model": "gpt-3.5-turbo",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

And here's how we can re-create this with the Julia HTTP library:

OPENAI_API_KEY = ENV[ "OPENAI_API_KEY"]

 using HTTP
 using JSON

headers = HTTP.Headers([
     "Authorization" =>  "Bearer $OPENAI_API_KEY",
     "Content-Type" =>  "application/json",
])

body = json(Dict( "model" =>  "gpt-3.5-turbo",
                  "messages" => [Dict( "role" =>  "user",  "content" =>  "Hello!")]))


response = HTTP.post(
     "https://api.openai.com/v1/chat/completions",
    headers,
    body;
    verbose =  false,
)

 #  Parse the response body as JSON
result = JSON.parse(String(response.body))
print(result)

Dict{String, Any}("choices" => Any[Dict{String, Any}("finish_reason" => "stop", "message" => Dict{String, Any}("role" => "assistant", "content" => "\n\nHello there! How may I be of assistance?"), "index" => 0)], "model" => "gpt-3.5-turbo-0301", "usage" => Dict{String, Any}("completion_tokens" => 12, "total_tokens" => 21, "prompt_tokens" => 9), "id" => "chatcmpl-6qMM6OmdVRZ8VtdKgKFtb7aRDYWkF", "object" => "chat.completion", "created" => 1677937010)

Note that OPENAI_API_KEY is stored as an environment variable on my system, so I was able to access it with OPENAI_API_KEY = ENV["OPENAI_API_KEY"].

We can extract the completion itself from the "choices" dictionary entry.

chat_response = result[ "choices"]
chat_response[1][ "message"][ "content"]

"\n\nHello there! How may I be of assistance?"

One of the things that makes ChatGPT useful is its ability to "remember" past parts of a conversation. We can send whole conversations to the API using the "messages" part of the request body. messages is an array of ~Dict~s, each of which has a "role" and a "content." There are three options for "role":

1. system: sets the behavior of the assistant. The content might be, for example, you are a helpful assistant or you are a very polite customer support agent or you are a senior software engineer in a mentorship role.
2. user: The human interacting with ChatGPT.
3. assistant: the responses from ChatGPT

So we can send a more complete conversation and get some richer details in response. For example:

messages=[Dict( "role" =>  "system",  "content" =>  "You are a knowledgable and helpful Julia developer."),
         Dict( "role" =>  "user",  "content" =>  "Can you show me how to make a POST request with the HTTP library?")]

body = json(Dict( "model" =>  "gpt-3.5-turbo",
                  "messages" => messages))

response = HTTP.post(
     "https://api.openai.com/v1/chat/completions",
    headers,
    body;
    verbose =  false,
)

 #  Parse the response body as JSON
result = JSON.parse(String(response.body))
print(result[ "choices"][1][ "message"][ "content"])

```julia
using HTTP

# URL to POST to
url = "https://httpbin.org/post"

# Data to include in the POST request (in JSON format)
data = Dict("name" => "John", "age" => 30)
json_data = JSON.json(data)

# Headers to specify that we're sending JSON data
headers = Dict("Content-Type" => "application/json")

# Make the POST request
response = HTTP.request("POST", url, headers, json_data)

# Get the response body as a string
body = String(response.body)

# Print the response status code and body
println("Status code: $(response.status)")
println("Response body: $body")
```

In this example, we're sending a JSON object with a name and age property to https://httpbin.org/post, which is an HTTP testing service. The `headers` argument specifies that we're sending JSON data, while the `json_data` argument is the actual data we want to send.

The `HTTP.request` function is called with the POST method, the URL to POST to, the headers we want to send, and the data we want to include. The response is then captured in the `response` variable.

Finally, we extract the body of the response as a string using `String(response.body)`, and print both the status code and response body to the console.Yes, I can. Here's an example code snippet that shows how to make a POST request using the HTTP library in Julia:

```julia
using HTTP

url = "https://jsonplaceholder.typicode.com/posts"
data = "{\"title\":\"foo\",\"body\":\"bar\",\"userId\":1}"

response = HTTP.post(url, data, ["Content-Type" => "application/json"])
println(String(response.body))
```

In this example, we first specify the url of the endpoint we want to send our request to. Next, we create a string representation of the JSON data we want to send in our request. We use `HTTP.post()` to send a POST request to the specified url, including the JSON data in the request body, and with a content type header that specifies that the data is JSON (application/json). Finally, we print the response contents converted to a string by the `String()` function.

And if we want to send a followup referring to an earlier part in the conversation, we can extend the messages array as follows:

 #  include the assistant's previous response
push!(messages, result[ "choices"][1][ "message"])

 #  ask a new question referring to an earlier part of the conversation
push!(messages, Dict( "role" =>  "user",
                      "content" =>  "Can you please provide a shorter, simpler example?"))

4-element Vector{Dict{String, String}}:
 Dict("role" => "system", "content" => "You are a knowledgable and helpful Julia developer.")
 Dict("role" => "user", "content" => "Can you show me how to make a POST request with the HTTP library?")
 Dict("role" => "assistant", "content" => "Yes, I can. Here's an example code snippet that shows how to make a POST request using the HTTP library in Julia:\n\n```julia\nusing HTTP\n\nurl = \"https://jsonplaceholder.typicode.com/posts\"\ndata = \"{\\\"title\\\":\\\"foo\\\",\\\"body\\\":\\\"bar\\\",\\\"userId\\\":1}\"\n\nresponse = HTTP.post(url, data, [\"Content-Type\" => \"application/json\"])\nprintln(String(response.body))\n```\n\nIn this example, we first specify the url of the endpoint we want to send our request to. Next, we create a string representation of the JSON data we want to send in our request. We use `HTTP.post()` to send a POST request to the specified url, including the JSON data in the request body, and with a content type header that specifies that the data is JSON (application/json). Finally, we print the response contents converted to a string by the `String()` function.")
 Dict("role" => "user", "content" => "Can you please provide a shorter, simpler example?")

And then request another response:

body = json(Dict( "model" =>  "gpt-3.5-turbo",
                  "messages" => messages))

response = HTTP.post(
     "https://api.openai.com/v1/chat/completions",
    headers,
    body;
    verbose =  false,
)

 #  Parse the response body as JSON
result = JSON.parse(String(response.body))
print(result[ "choices"][1][ "message"][ "content"])

Sure, here's a simpler example:

```julia
using HTTP

url = "https://jsonplaceholder.typicode.com/posts"

response = HTTP.post(url, form = [("title", "foo"), ("body", "bar"), ("userId", "1")])
println(String(response.body))
```

In this example, we are sending a POST request with form data rather than JSON data. We specify the form data using an array of tuples with the keys and values for the form fields. Note that we use the `form` argument to pass the form data to `HTTP.post()`. The response is then printed in the same way as before.

This post showed basic usage of the ChatGPT API with Julia. In the next post, I'll show how to make this more modular and useful. We'll create a Struct for conversations and a function to call on the API based on the conversation history in that Struct. After that, I'll write about how to make it interactive, perhaps as a Julia REPL mode, but at least as a command line utility.

Find the code snippet here.

All the details about creating a custom Denote filetype can be found here. First, let's inspect the denote-file-types alist to understand what a file type definition looks like.

(print denote-file-types)

((quarto :extension ".qmd" :date-function denote-date-iso-8601 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (quarto :extension ".qmd" :date-function denote-date-rfc3339 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (org :extension ".org" :date-function denote-date-org-timestamp :front-matter denote-org-front-matter :title-key-regexp "^#\\+title\\s-*:" :title-value-function identity :title-value-reverse-function denote-trim-whitespace :keywords-key-regexp "^#\\+filetags\\s-*:" :keywords-value-function denote-format-keywords-for-org-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-org-link-format :link-in-context-regexp denote-org-link-in-context-regexp) (markdown-yaml :extension ".qmd" :date-function denote-date-rfc3339 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (markdown-toml :extension ".md" :date-function denote-date-rfc3339 :front-matter denote-toml-front-matter :title-key-regexp "^title\\s-*=" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*=" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (text :extension ".txt" :date-function denote-date-iso-8601 :front-matter denote-text-front-matter :title-key-regexp "^title\\s-*:" :title-value-function identity :title-value-reverse-function denote-trim-whitespace :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-text-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-org-link-format :link-in-context-regexp denote-org-link-in-context-regexp))

We need to add to this list. We can define a new filetype in this alist by defining a new element of the form (SYMBOL PROPERTY-LIST).

Adding a basic quarto file should be very simple. We only need to duplicatge the markdown file type definition, changing the name to quarto and the extension to ".qmd". We could do this as follows.

(add-to-list 'denote-file-types
             '(quarto
                :extension  ".qmd"
                :date-function denote-date-iso-8601
                :front-matter denote-yaml-front-matter
                :title-key-regexp  "^title\\s-*:"
                :title-value-function denote-surround-with-quotes
                :title-value-reverse-function denote-trim-whitespace-then-quotes
                :keywords-key-regexp  "^tags\\s-*:"
                :keywords-value-function denote-format-keywords-for-md-front-matter
                :keywords-value-reverse-function denote-extract-keywords-from-front-matter
                :link denote-md-link-format
                :link-in-context-regexp denote-md-link-in-context-regexp))

((quarto :extension ".qmd" :date-function denote-date-iso-8601 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (quarto :extension ".qmd" :date-function denote-date-rfc3339 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (org :extension ".org" :date-function denote-date-org-timestamp :front-matter denote-org-front-matter :title-key-regexp "^#\\+title\\s-*:" :title-value-function identity :title-value-reverse-function denote-trim-whitespace :keywords-key-regexp "^#\\+filetags\\s-*:" :keywords-value-function denote-format-keywords-for-org-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-org-link-format :link-in-context-regexp denote-org-link-in-context-regexp) (markdown-yaml :extension ".qmd" :date-function denote-date-rfc3339 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (markdown-toml :extension ".md" :date-function denote-date-rfc3339 :front-matter denote-toml-front-matter :title-key-regexp "^title\\s-*=" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*=" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (text :extension ".txt" :date-function denote-date-iso-8601 :front-matter denote-text-front-matter :title-key-regexp "^title\\s-*:" :title-value-function identity :title-value-reverse-function denote-trim-whitespace :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-text-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-org-link-format :link-in-context-regexp denote-org-link-in-context-regexp))

We can do this more concisely (and learn a litle bit of emacs lisp on the way) by modifying the markdown-yaml file type definition.

As noted, all we're actually doing is changing the file extension in the existing markdown type. So instead of writing out all of the redundant details, let's just copy them from the markdown type.

( let ((quarto (cdr (assoc 'markdown-yaml denote-file-types))))
  ( setf (plist-get quarto  :extension)  ".qmd")
  (add-to-list 'denote-file-types (cons 'quarto quarto)))

(print denote-file-types)

((quarto :extension ".qmd" :date-function denote-date-rfc3339 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (org :extension ".org" :date-function denote-date-org-timestamp :front-matter denote-org-front-matter :title-key-regexp "^#\\+title\\s-*:" :title-value-function identity :title-value-reverse-function denote-trim-whitespace :keywords-key-regexp "^#\\+filetags\\s-*:" :keywords-value-function denote-format-keywords-for-org-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-org-link-format :link-in-context-regexp denote-org-link-in-context-regexp) (markdown-yaml :extension ".qmd" :date-function denote-date-rfc3339 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (markdown-toml :extension ".md" :date-function denote-date-rfc3339 :front-matter denote-toml-front-matter :title-key-regexp "^title\\s-*=" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*=" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp) (text :extension ".txt" :date-function denote-date-iso-8601 :front-matter denote-text-front-matter :title-key-regexp "^title\\s-*:" :title-value-function identity :title-value-reverse-function denote-trim-whitespace :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-text-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-org-link-format :link-in-context-regexp denote-org-link-in-context-regexp))

What's actually happening here? Let's go through it step by step.

The assoc function takes a key and an alist as arguments and returns the first element of that alist whose CAR is equal to the key. In effect, we're searching the denote-file-types alist for the element whose CAR is markdown-yaml.

(assoc 'markdown-yaml denote-file-types)

(markdown-yaml :extension ".qmd" :date-function denote-date-rfc3339 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp)

We only want the CDR of this element (everything but the markdown-yaml at the beginning).

(cdr (assoc 'markdown-yaml denote-file-types))

(:extension ".qmd" :date-function denote-date-rfc3339 :front-matter denote-yaml-front-matter :title-key-regexp "^title\\s-*:" :title-value-function denote-surround-with-quotes :title-value-reverse-function denote-trim-whitespace-then-quotes :keywords-key-regexp "^tags\\s-*:" :keywords-value-function denote-format-keywords-for-md-front-matter :keywords-value-reverse-function denote-extract-keywords-from-front-matter :link denote-md-link-format :link-in-context-regexp denote-md-link-in-context-regexp)

We use let to say that, for the purposes of this procedure, we want to associate the above plist with the name quarto.

Next, we need to change the extension from .md to .qmd. We use plist-get to get the desired property ( :extension) and setf to set it to ".qmd".

( setf (plist-get (cdr (assoc 'markdown-yaml denote-file-types))  :extension)  ".qmd")

.qmd

In the end, we add the quarto plist to the denote-file-types alist with (add-to-list 'denote-file-types (cons 'quarto quarto)). The cons function creates a new cons from a CAR and a CDR (in this case, quarto and the modified copy of the markdown-yaml plist with the changed extension).

And with that, we can now create new quarto files with denote!

Check out quarto-emacs if you're interested in working with quarto files in emacs (though I have to admit that working with quarto files in RStudio is a joy and is at least worth a try before commiting to an emacs-only workflow).

In this example, we'll download some data from the Bureau of Labor Statistics API, apply some minor processing, and save the results as a CSV. More specifically, we will:

1. Download two BLS data series from the BLS API using curl
2. Extract a few different variables from different hierarchical levels of the JSON results using jq
3. Save the results as a CSV using jq

This mirrors a common Python task in my workflow: I make an API call using the requests module, load the JSON results as a Python dict type with json.load from the json module, and then load the results as a pandas table with Pandas.DataFrame.from_dict. There may be more (or fewer) steps depending on the structure of the API results, but I've repeated this broad pattern many times.

To follow along, begin by installing jq if you haven't already.

As noted above, we use curl to make a POST request agains the BLS API. We're downloading Consumer Price Index data since the start of 2022. We're downloading data from two BLS series: one for seasonally adjusted data, and one for unadjusted data.

curl -X POST -H  'Content-Type: application/json'  \
     -d  '{"seriesid": ["CUUR0000SA0","CUSR0000SA0"], "startyear":2022, "endyear":2022}'  \
     https://api.bls.gov/publicAPI/v2/timeseries/data/ 

| status":"REQUEST_SUCCEEDED","responseTime":171,"message":[],"Results               |
| series                                                                            |
| seriesID":"CUUR0000SA0","data":[{"year":"2022","period":"M08","periodName":"August", "latest":"true","value":"296.171","footnotes":[{}]},{"year":"2022","period":"M07","periodName":"July","value":"296.276","footnotes":[{}]},{"year":"2022","period":"M06","periodName":"June","value":"296.311","footnotes":[{}]},{"year":"2022","period":"M05","periodName":"May","value":"292.296","footnotes":[{}]},{"year":"2022","period":"M04","periodName":"April","value":"289.109","footnotes":[{}]},{"year":"2022","period":"M03","periodName":"March","value":"287.504","footnotes":[{}]},{"year":"2022","period":"M02","periodName":"February","value":"283.716","footnotes":[{}]},{"year":"2022","period":"M01","periodName":"January","value":"281.148","footnotes |
| seriesID":"CUSR0000SA0","data":[{"year":"2022","period":"M08","periodName":"August", "latest":"true","value":"295.620","footnotes":[{}]},{"year":"2022","period":"M07","periodName":"July","value":"295.271","footnotes":[{}]},{"year":"2022","period":"M06","periodName":"June","value":"295.328","footnotes":[{}]},{"year":"2022","period":"M05","periodName":"May","value":"291.474","footnotes":[{}]},{"year":"2022","period":"M04","periodName":"April","value":"288.663","footnotes":[{}]},{"year":"2022","period":"M03","periodName":"March","value":"287.708","footnotes":[{}]},{"year":"2022","period":"M02","periodName":"February","value":"284.182","footnotes":[{}]},{"year":"2022","period":"M01","periodName":"January","value":"281.933","footnotes |
| }}                                                                                |

This is technically readable, but not very nice.

For most use cases, the data aren't particularly usable in this format. They're hard to read; they include metadata about the API response; and the hierarchical structure makes them difficult to extract into a tabular format. Enter jq. We'll proceed step by step to show how we can use jq to extract specific data from the API results.

If we use the default settings, figures aren't wrapped in

To address this, we need to change the org-html-doctype and org-html-html5-fancy variables. We can do this in the export configuration's org-publish-project-alist. See here for the updated configuration file. Here is what the updated org-publish-project-alist looks like—see lines 21-22 for the key changes.

  1: ( setq org-publish-project-alist
  2:       (list
  3:        (list  "org-site:main"
  4:               :recursive t
  5:               :base-directory  "./content"
  6:               :publishing-directory  "./public"
  7:               :publishing-function 'org-html-publish-to-html
  8:               :html-preamble (file-contents  "assets/html_preamble.html")
  9:               :with-author nil
 10:               :with-creator t
 11:               :with-toc t
 12:               :section-numbers nil
 13:               :time-stamp-file nil
 14:               :auto-sitemap t
 15:               :sitemap-title nil ; "Daniel Liden's Blog"
 16:               :sitemap-format-entry 'my/org-publish-org-sitemap-format
 17:               :sitemap-function 'my/org-publish-org-sitemap
 18:               :sitemap-sort-files 'anti-chronologically
 19:               :sitemap-filename  "sitemap.org"
 20:               :sitemap-style 'tree
 21:               :html-doctype  "html5"
 22:               :html-html5-fancy t)
 23:        (list  "org-site:static"
 24:               :base-directory  "./content/"
 25:               :base-extension  "css  \\  | js  \\  | png  \\  | jpg  \\  | gif  \\  | pdf  \\  | mp3  \\  | ogg  \\  | swf  \\  | svg"
 26:               :publishing-directory  "./public"
 27:               :recursive t
 28:               :publishing-function 'org-publish-attachment
 29:              )
 30:        (list  "org-site:assets"
 31:               :base-directory  "./assets/"
 32:               :base-extension  "css  \\  | js  \\  | png  \\  | jpg  \\  | gif  \\  | pdf  \\  | mp3  \\  | ogg  \\  | swf"
 33:               :publishing-directory  "./public/"
 34:               :recursive t
 35:               :publishing-function 'org-publish-attachment)))

With these changes, our images and captions are now centered:

And the html generated from the org export now looks like this:

< figure  id= "orgbffad64">
< img  src= "figures/20220719-julia-plots/fig5.png"  alt= "fig5.png"  width= "450px">

< figcaption>< span  class= "figure-number">Figure 1: span>This caption is now centered and wrapped in a < code> <figcaption >code> tag!figcaption>
figure>

We now have a

While julia-snail is my favorite Julia development environment, it's support for org-mode is, at present, quite limited. For example, it seems to ignore the :file argument, making it difficult to save figures to specific locations. I find that emacs-jupyter provides the most featureful and reliable way to work with Julia in org-mode. Read the emacs-jupyter documentation for instructions on how to use emacs-jupyter with org-mode.

Let's make some plots. We'll use the Plots.jl package and explore a few different plotting styles. First, we load the packages we'll be using. I tend to plot statistical distributions fairly often, so I'll load Distributions and Statplots in addition to Plots.

using Plots
using Distributions
using StatsPlots

That's enough of that. Usually we're plotting real data, not standard standard distributions. Let's get some. First we'll pull from the RDatasets package, which is an excellent source of go-to data science and statistics examples such as mtcars and iris. We'll use the venerable mtcars to show how to work with data in a basic way.

using RDatasets, DataFrames
mtcars = dataset("datasets", "mtcars")
mtcars[1:10,:]

We can use the @df macro and access columns by prepending them with :.

gr()
@df mtcars scatter(:MPG, :HP, group=:Cyl, background=:black, msize=8, keytitle="N Cylinders")
xlabel!("Mpg")
ylabel!("HP")
title!("Horsepower by MPG and N Cylinders")

It's not always straightforward to figure out how to modify visual components of a plot. Most of the relevant information lies in the Attributes section of the Plots.jl documentation (and its subsections).

One of the benefits of the Plots.jl package is its support for many different plotting backends. The default is GR which, according to the documentation, is

The default backend. Very fast with lots of plot types. Still actively developed and improving daily.

and it offers speed; 2D and 3D plots, and standalone or inline plotting.

Here we'll repeat the plot above with the UnicodePlots backend. We first need to install the package with add UnicodePlots in the package manager. Note that you can do this right from the Jupyter repl in emacs-jupyter by pressing ] in the repl.

We then specify that we'd like to use this backend with the unicodeplots() function.

unicodeplots()
# scatterplot
@df mtcars scatter(:MPG, :HP, title="HP vs MPG", xlabel="Mpg", ylabel="HP")

                        HP vs MPG                   
       +----------------------------------------+   
343.49 |        ⚬                               | y1
       |                                        |   
       |                                        |   
       |         ⚬                              |   
       |     ⚬ ⚬                                |   
       |        ⚬                               |   
       | ⚬                                      |   
    HP |                                        |   
       |        ⚬ ⚬ ⚬ ⚬⚬⚬                       |   
       |        ⚬⚬                              |   
       |                                        |   
       |             ⚬ ⚬  ⚬              ⚬      |   
       |                  ⚬  ⚬    ⚬             |   
       |                            ⚬       ⚬ ⚬ |   
 43.51 |                       ⚬         ⚬      |   
       +----------------------------------------+   
        9.695             Mpg             34.605    

I haven't yet figured out how to get the colors to appear as they are supposed to in org mode, so for now, I'm simply showing a basic scatterplot with no grouping. I'll update if and when I figure that out. It should look like this:

There's a lot more to get into with plotting in Julia and with using Julia in emacs. This post serves as a small jumping-off point—just enough to get started, with a few pointers to further resources, and some questions to start pursuing. I'll write more on this topic as I learn more!

Here is an example of the issue. Below, I inserted an org source block (defined with #+begin_src org) with a first-level org headline inside, without inserting a comma before the headline.

#+begin _src org

And it is still parsed and exported as a headline, not as an example. #+end _src

Clearly, this isn't what we want.

This issue is easily resolved: just insert a comma before the headline in the source block (e.g. ,* Headline). This results in the following:

 * This Headline Is in an Org Source Block
and it looks like org source, not like an exported org file.

This approach works well with more complicated org-mode syntax, as well. For example, the following block:

,* Headline 1
:PROPERTIES:
:ID:       642BF4EE-3139-4B96-97C4-D3BABD86FFD5
:END:
,** Headline 2
This headline is also prepended with a comma!
,** Even a Nested Source Block!
,#+begin_src R
1+1
,#+end_src

,#+RESULTS:
: 2

leads to the following (when we specify that it is an org source block).

 * Headline 1
 :PROPERTIES:
 :ID:        642BF4EE-3139-4B96-97C4-D3BABD86FFD5
 :END:
 ** Headline 2
This headline is also prepended with a comma!
 ** Even a Nested Source Block!
 #+begin_src R
 1+1
 #+end_src

 #+RESULTS:
 : 2

We can use the org-edit-special command ( C-c ') within an org source block to open a separate editing environment (in this case, another org buffer). We can then write some org syntax. When we exit the environment (again, with C-c '), it will be properly formatted for export.

The following block was generated using this method:

,* Headline 1                                                    :example_tag:
This is an org headline with a tag.
,** TODO Second Headline
SCHEDULED: <2022-02-15 Tue>
:PROPERTIES:
:CUSTOM_ID: headline_1
:END:
,** Babel

,#+begin_src jupyter-python :session datasci
import numpy as np

np.array(range(1,10))
,#+end_src

,#+RESULTS:
: array([1, 2, 3, 4, 5, 6, 7, 8, 9])

Which generates:

 * Headline 1                                                      :example_tag:
This is an org headline with a tag.
 **   TODO  Second Headline
 SCHEDULED:  <2022-02-15 Tue>
 :PROPERTIES:
 :CUSTOM_ID:  headline_1
 :END:
 ** Babel

 #+begin_src jupyter-python :session datasci
 import numpy as np

np.array(range(1,10))
 #+end_src

 #+RESULTS:
 : array([1, 2, 3, 4, 5, 6, 7, 8, 9])

This approach is easier because there is no need to remember or guess what, exactly, needs to be prepended with a comma. In this case, the headlines themselves, the nested #+begin_src and #+end_src lines, and the #+RESULTS: line were all escaped with commas.

With both of these approaches available, writing about org mode with example org source blocks should be much easier.

The org-mode manual provides plenty of details on the basics of dates and times. Here's a quick review:

• Add a timestamp with only the date to an org-mode entry with .. A timestamp entry with a time can be added with .. While there are plenty of methods for modifying timestamps after creation, I typically create the timestamp and then modify it, if needed, with S-LEFT/RIGHT/UP/DOWN to change the day (forward/back) or whichever element of the timestamp the cursor is located on (up/down).

  Another useful timestamp format is ! (org-time-stamp-inactive), which inserts an inactive timestamp (represented with square rather than angled brackets; e.g. [2022-01-16 Sun] rather than <2022-01-16 Sun>). Inactive timestamps work the same as active timestamps, except that they do not appear on the org-agenda.

• Two timestamps connected by -- make up a range (e.g. <2022-02-12 Sat>--<2022-01-16 Sun>). A range will appear on the agenda on the start and stop dates and on all dates in the range.

• Add a scheduled date to a headline with and a deadline with . Again, there are plenty of details in the manual. Both of these types of timestamps affect how the timestamp appears in the agenda. In short, a "scheduled" headline appears on the date of the timestamp and on every day following until the entry is marked DONE. A "deadline" timestamp on a headline will start appearing in the agenda org-deadline-warning-days before the deadline timestamp and will appear daily until it is marked DONE.

  Conceptually, a "deadline" in org mode comports with the common use of the word: it is the date when a task is supposed to be finished. The meaning of "scheduled," on the other hand, specifically refers to the date when a task is to be started. It is not meant to be used in the sense of "scheduling an appointment." A regular timestamp is better suited for this use. Recall: a scheduled headline will appear in the agenda even after the scheduled date, which is not especially useful for an event (again, like a meeting or appointment) that occurs on a given date.

Org-mode makes it possible to schedule recurring events without manually specifying each repetition date. There are three key formats for repeated events. Regardless of the format, the main idea is to add a repetition interval to the timestamp. A repetition interval combines a number with a unit of time to specify how often a timestamp/schedule/deadline should be repeated. For example:

• <2022-01-16 Sun +1d> will repeat daily.
• <2022-01-29 Sat +5d> will repeat every five days.
• <2022-01-29 Sat +1y> will repeat yearly.

When specifying repeat intervals, y means "yearly", m means "monthly", w means "weekly", d means "daily", and h means "hourly." Prepending these values with + tells how often the event should be repeated.

The org-mode manual is the best place to learn more about timestamps, schedules, and deadlines in general and repeated tasks in particular. Here are some places to start:

• Timestamps
• Repeated Tasks
• Dates and Times (Index Page)

I will specifically talk about using org-babel in interactive (session-based) evaluation, much like how you would use a Jupyter notebook. Each org-babel source block is executed in the same environment, so data objects persist between source blocks. Some specific org-babel behaviors will depend on the specifics of your org-babel configuration; you can see mine below ¹.

For notes on setting up org-babel to work with R, read this article.

I tend to use two different header configurations, one for code output and one for figures.

The excellent emacs-jupyter package is a solid alternative to the approaches described above. To use it for R, install the emacs-jupyter package (e.g. with (use-package jupyter). In R, install IRkernel (make sure to follow all of the instructions here for registering the kernel). Update org-babel-load-languages to include jupyter. E.g.:

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (julia . t)
   (python . t)
   (jupyter . t)))

Then you can set up R blocks with:

#+begin_src jupyter-R :session my-jupyter-session

e.g.

1+1

[1] 2

Using emacs-jupyter results in a significantly different experience, which warrants a post of its own (especially because the documentation for using emacs-jupyter with R is not easy to find or follow).

A couple of reasons emacs-jupyter is worth considering:

• if a code block generates a figure, you don't need to specify a filename/path or use any fancy header arguments to make sure the image appears in your org buffer. The image is named and saved automatically to org-babel-jupyter-resource-directory. This makes exploratory analysis comparatively seamless. I spend much less time tripping over header args with emacs-jupyter.
• Accessibility of help: pressing with the cursor over an object opens a new window with documentation.

There were a few simple design and organization changes I wanted before I started writing: a simple and readable theme; a list of recent posts with short "preview" snippets, and a link to a post archive.

Check out the full source code for the site on GitHub.

This site now has everything I need to write more with minimal need to tweak the site configuration every step of the way (which I found myself doing constantly when using Hugo). That said, there are a few more things I want to do before moving this site to its permanent home under my personal domain. These include:

Good resoures for setting up a custom sitemap as an index page.

Nice-looking basic stylesheet for site. I likely won't stick with this forever but it looks good for now.

This site includes a good overview of making list of posts with previews of the first few lines of text.

By Dennis Ogbe. The code I ultimately used for generating and showing previews closely followed that from this post.

Good tutorial covering the basics of publishing org-mode projects to html.

Here is a link to the site I'm making. Ultimately I will add a custom domain.

This short write up is in pursuit of a personal goal to put more of my work and thoughts to paper. I'm sure the topic will still be useful to others, but it's not yet fully developed. It's a sketch of a work in progress.

Last August, I took the fast.ai Practical Deep Learning for Coders online course (and worked through the accompanying book). The course taught the fundamentals of deep learning using the fastai Python library, which offers a higher-level API (and a vast range of useful features and tools) for PyTorch. The course (and book) followed a "top-down" approach: learn how to effectively apply the models first, and then go back and learn the more foundational concepts, math, etc., in greater detail.

After spending several months using fastai for a number of tasks (including the Kaggle Cassava Leaf Disease Classification competition, the numer.ai tournament, and a turtle classifier), I decided I wanted to "pull back the curtain" and start to learn how to use PyTorch. The numer.ai tournament seemed like an excellent opportunity to do so. The tournament data come in a very "clean" and ready-to-use format, so grappling with the data doesn't have to be a huge part of the modeling process. The dataset is big enough for deep learning but not so huge that the models can't be run locally. And I already have a working fast.ai model up and running, and I know it's running PyTorch under the hood, so I know that it will work!

One quick note: trying to learn PyTorch is inspired by a desire to learn more; not by any serious perceived weaknesses in fastai. Many fastai users and community members, such as Zachary Mueller (see the excellent walk with fastai project) have shown that fastai is extremely flexible and extensible. That said:

• Implementations of new methods often appear first as PyTorch models. Using them directly is easier to me than translating them to a fastai context.
• fastai sometimes hides so many details that I find it hard to determine what, exactly, my models are doing. For example, it took me a while to discover that the fastai tabular model I was using for the Numerai tournament was implementing batchnorm and dropout layers. The architecture of the model was, at least initially, opaque to me.
• While fastai allows plenty of flexibility, it isn't necessarily built for it. I often need to do quite a bit of research to figure out how to tweak my training loop in a way that would be trivial in a direct PyTorch implementation.
• Some of the other learning resources I've been using, such as the excellent Dive into Deep Learning book, use PyTorch. Rather than translating their code into fastai, I would prefer to learn PyTorch directly.

In this post, I will first briefly show the fastai model I was previously using, and then introduce a (simpler) model written in PyTorch. I will conclude with some reflections on the process.

My original fastai implementation does not differ appreciably from the implementation detailed in the official fastai Tabular Training tutorial. The components are, briefly:

In an effort to learn some basic PyTorch, I set out to develop a very simple working model. It doesn't have all of the bells and whistles of the fastai model – no batchnorm, no dropout, no weight decay – but it works and it is generally easy to understand what the model is doing. This provides a good foundation for further experimentation with more complex architectures.

I wrote a lot more code to implement a comparatively-simple PyTorch model than to implement the fastai model. The PyTorch model forces us to better understand the structure of the model and the logic of the training loop, though it likely takes more time and more finessing to obtain an efficiently-performing model with decent results. The fastai model, on the other hand, is very quick to implement but does not expose as many of the details. It is relatively quick and easy to get a model running and returning decent results, but it can take a bit more work to understand the structure of the model and of the training loop.

I'll be writing more – and more complicated – PyTorch models in the future. I hope to add in some of the additional features included in the fastai tabular implementation, such as dropout layers. I also want to experiment further with regularization – L2 penalization is very easy to use in PyTorch, but I've found L1 penalization to work far better for regression models in the Numerai tournament and I want to see if that distinction also holds true for regression models.

For the past several months, I have been working on an analysis on the effects of urban heat on vulnerable populations, particularly during a public health crisis. For some background, I currently live in Las Vegas, where summer heat can exceed 110 degrees. Last summer included a 45-day-long streak of temperatures over 100 degrees.

Urban heat is not distributed evenly within cities. Features such as parks or ponds can lead to cooler temperatures in some areas, while areas without foliage or with dark surfaces such as roads and buildings lead to warmer temperatures. This effect is particularly pronounced at night: urban surfaces absorb heat during the daytime, warming the air at night.

An NPR Special Series titled Heat and Health in American Cities compared urban heat to household income in the 100 most populated cities. They found that, in many cities, income and urban heat were inversely related: lower-income households tended to be in the parts of cities most affected by urban heat, while higher-income households were more often found in cooler areas.

The key to this research was aggregating urban heat data within census boundaries (such as tracts or blocks). I decided to recreate this analysis in R. Once urban heat data is linked to Census-designated boundaries, it can easily be compared to metrics such as income, poverty, disability, access to health insurance, demographics, any many other data points collected by the U.S. Census Bureau's American Community Survey (ACS). In R, obtaining and mapping these metrics is a relatively simple process with packages such as tidycensus and tigris.

The remainder of this post will provide a very brief overview of how to obtain satellite surface temperature data; aggregate those data points within census tracts; and plot the results. I don't know if this mapping exercise will make it into the final project report, but I learned a lot from working through it, and I wanted to make sure to document my steps here.

Though it should be possible to follow this guide without much prior experience working with spatial data, This free book, Geocomputation with R by Robin Lovelace, Jakub Nowosad, Jannes Muenchow, is an excellent resource for those interested in learning more. It is particularly useful for understanding topics such as projections and the differences between raster and vector data. The section on Raster-vector interactions is particularly relevant to this task as we will be taking raster values (the heat map) and aggregating them within polygon boundaries (census tracts).

I obtained surface temperature satellite data from the NASA/USGS Landsat 8 satellite. The data can be obtained through an API, though for this example, I downloaded the data manually. I looked for data meeting the following criteria:

• From June-August 2020
• Cloud cover less than 4%
• Landsat C1 Analysis-Ready Data (ARD) datasets
• I ultimately downloaded a raster map with ID LC08_CU_005011_20200806_20200824_C01_V01. This map came from 06 August, 2020. Details, including a download link, can be found here.

So what data did we actually obtain? Downloading the "Provisional Land Surface Temperature" file returns a .tar file with a number of .tif raster images. We're interested in the one with ST.tif at the end (ST for Surface Temperature). This provides a raster map of the region, at 30-meter spatial resolution, with surface temperatures in tenths of a degree Kelvin. That is, each 30-meter "cell" in the map is associated with a surface temperature value.

First, we'll use the tigris package to download the census tract boundaries in Nevada and crop to the Las Vegas area.

lv_tracts  <- tigris::tracts(state= "NV")  %>%
  st_crop(xmin=-115.38, ymin=35.92, xmax = -114.88, ymax = 36.38) 

Next, we'll reproject our tract-level data such that it has the same projection as the raster data. Computationally, transforming vector data (our tract data) is much less expensive than transforming raster data. After reprojecting, we can use the crop function from the raster package to crop the raster image to the Las Vegas area as defined in lv_tracts.

lv_tracts_reprojected = st_transform(lv_racts, crs(st_map))
sat_cropped = crop(st_map, lv_tracts_reprojected)

With this accomplished, we use the raster::extract() function to extract the mean of the raster values within the boundaries defined by lv_tracts_reprojected, which contains the census tract boundaries projected to align with the raster map.

heat_map = extract(sat_cropped,            #  cropped raster object
                   lv_tracts_reprojected,  #  vector map of LV
                   df= TRUE,                #  return as data frame
                   fun=mean,               #  return the mean of each polygon
                   sp= TRUE)                #  append to lv_tracts_reprojected

 ##  Visualize
sf_heat_map = st_as_sf(heat_map)           #  Convert to simple features (sf) vector data
png(file= "IMAGE_DESTINATION/heatmaptest.png")
plot(out[ "st_map"])                        #  plot the column of raster values
dev.off()

We're closer to our goal, but not quite there yet. Our map is rotated, and the scale (tenths of a degree Kelvin) isn't especially interpretable. First, we'll reproject the map to the correct orientation (back to the original projection of the census tract data).

heat_tracts_transformed = st_transform(x=sf_heat_map, crs = crs(lv_tracts))
png( "tract_repro.png")
plot(heat_tracts_transformed[ "st_map"])
dev.off()

Lastly, we can apply some formatting to make it more interpretable. I used ggplot2 for all of the visual tweaks (details not shown).

We can now plot other metrics by census tract and visually compare them to our urban heat map. For example, we can look at poverty by census tract:

A quick visual inspection of these two maps shows that the Sunrise Manor and Winchester areas have relatively high temperatures and a high proportion of residents living abelow the poverty line. Conversely, the Summerlin area on the west side of Las Vegas has among the lowest temperatures and the lowest rates of poverty.

There are plenty of additional analyses we can conduct from here. The NPR report linked above calculated correlations between tract-level surface temperatures and household incomes to determine that the two were inversely correlated. There are also a variety of spatial regression models and statistical learning/machine learning techniques that can be applied to spatial data. Understanding how to connect different sources of spatial data – such as the census tracts and heat data above – is an important first step to conducting these analyses.

• CRAN Task View: Analysis of Spatial Data: A CRAN hub explaining the R ecosystem of spatial data analysis packages
• Spatial Data Science with R: A site providing a broad overview of spatial data science concepts and methods in R.
• Geocomputation with R: A book by Robin Lovelace, Jakub Nowosad, and Jannes Muenchow, now published by CRC press. I learned most of what I presented above from this book, which provides a thorough account of the ways of manipulating, mapping, and analyzing spatial data, with numerous excellent examples.

In this post, I am briefly reviewing the draft of Andrew Ng's Machine Learning Yearning, which can currently be obtained for free here. This is a work in progress, so my comments will be brief and not particularly critical. I obtained the copy I read on May 11; the page numbers and content may have changed since then.

I read the book because I have not had the chance to do too much machine learning work in the recent past. I feared that my skills were getting rusty, so I wanted to "dip my toes" back in the ML water and refresh my memory of that world. This is an ongoing process.

Most of my ML education comes from working through much of the classic Elements of Statistical Learning during my Statistics MS, so my perspective is significantly influenced by my experience with that book.

In Machine Learning Yearning, Ng states that "after finishing this book, you will have a deep understanding of how to set technical direction for a machine learning project" (pg. 8). The key element of that statement is technical direction. This book does not teach (and does not claim to teach) any particular ML algorithms. There is no code. There is very, very little mathematical notation. This book covers broad topics such as structuring training/dev/test sets and conducting error analyses, and it does so in a way that is largely agnostic to the ML algorithms used (though, at times, it is fairly clear that it is geared largely toward deep learning/neural networks).

As of May 11, 2019, the book is divided into 58 chapters. Most chapters are only a page or two long. Ng states that the brevity of the chapters is such that "you can print them out and get your teammates to read just the 1-2 pages you need them to know." The short chapters did make the book easy to read as it divided the content into (very) short and easily-digestible pieces. However, I often did not find the chapters sufficiently self-contained to justify the divisions. I don't think the book would have suffered from having chapters 5-10 pages in length instead of 1-2 pages, and slightly longer chapters may have made the flow of information a little easier to follow (not that it was ever a significant challenge).

Core topics covered include:

• Composition of the train/dev/test sets (chapters 5-7, 11-2)
• Good characteristics of optimization metrics (chapters 8-12)
• Error analysis (chapters 14-19)
• Bias-Variance Trade-off; training vs. test error (chapters 20-32)
• Comparisons to human-level performance (chapters 33-35)
• Training/Testing on different data distributions; data mismatch errors; generalization from training to dev set (chapters 36-43)
• Optimization Verification Testing (chapters 44-46)
• End-to-end learning vs. pipeline learning (chapters 47-49)
• Pipeline learning: choosing components; error analysis by parts (chapters 50-57)

I found that this book provided a valuable supplement to my past reading and experience with machine learning insofar as it got "out of the weeds" and discussed the process rather than the specific implementations. When learning specific ML techniques for the first time, it's easy to get lost in the weeds. And when something goes wrong, that's where one might look for solutions: in the weeds. Is the implementation wrong? Do I need to improve my feature selection? Is the particular method I'm using wrong? This book offers another perspective. Regardless of the techniques used, it may be possible to improve one's model by, for example, seeking out more training examples; conducting a careful error analysis; or increasing the size/complexity of the model. These "big picture" considerations are often taught as an afterthought, if at all, in more mathematically-focused ML books and courses. But they should be taught sooner as they provide an excellent framework for thinking about any given ML approach.

The book naturally had some limitations. Once again, this is an early version; it is entirely possible that some or all of these issues will be addressed at some point in the future. I believe the book should have been clearer from the beginning that the focus was on deep learning/neural networks. Midway through the book, Ng notes that reducing the number of features is not recommended for reducing variance, as feature selection in general is de-emphasized in modern deep learning. Aside from this and the selection of examples, I don't think the deep learning focus is entirely clear. Furthermore, the prerequisites are a little vague. The book was very easy to follow, but it sometimes suffered from avoiding too much detail. Casual references to "regularization" or "early stopping," without more than a cursory explanation of those terms, may be confusing (or simply unnecessary) to some readers.

The book would benefit greatly from a set of simple but well-implemented examples, perhaps in R Shiny or in a Jupyter notebook, of the various key points made throughout the book. This could be done without expecting much, if any, programming experience on the part of the reader, and it would go a long way toward illustrating some of the concepts that may not be entirely intuitive. For example, the reader could explore the impact of trying to generalize a cat identifying tool from pictures found online to pictures taken from smartphones, or the difference in a sentiment analyzer based in an "end-to-end" vs. learning pipeline approach. The examples in the text are certainly useful, but the ability to make changes and explore the results would significantly enhance the experience.

The barriers to entry for ML are decreasing. Specific knowledge of the algorithms themselves is becoming less and less of a requirement (whether this should be the case is a different question, and one I will not attempt to address here). Books like this will be invaluable for guiding the next generation of ML practitioners, many of whom will benefit more from a strong understanding of the big-picture process than from a more theoretical understanding of the specific methods being used. This is a good book; many will find it useful.