Lot of new changes for observe this week. If you set the returnAction flag to true, it'll return the suggested Playwright method to run.

Also, Stagehand leverages the accessibility (a11y) tree to improve speed, accuracy, and costs fewer tokens. Using the a11y tree, however, means Stagehand may be able to act on items that aren’t directly visible (i.e. hidden behind a dropdown). To use these new features, set onlyVisible: false

const results = await page.observe({
     instruction: "click the sign in button",
     returnAction: true, // Return a Playwright action
     onlyVisible: false // Leverage the a11y tree (faster/better/cheaper)
})

// Act on an observation
// NO ADDITIONAL LLM CALL HERE! Just does the thing :)
await page.act(results[0]

This returns a list like this, which now contains method and arguments

[
  {
    description: 'The button saying sign in...'
    method: 'click', // or another playwright method
    arguments: [],
    selector: 'xpath=/html/body/div[1]/main/section/div[1]

Now, you can cache the selector, method, and arguments if you want to repeat the same Playwright without more LLM calls.

We also made some huge improvements to create-browser-app, making vercel AI SDK + ollama first-class citizens and greatly improving cursorrules.