Use an LLM - Inworld AI Documentation

The node-llm-chat template illustrates how to make LLM calls using the LLM node with support for streaming, tool calling, and multimodal inputs.

Architecture

Backend: Inworld Runtime
Frontend: N/A (CLI example)

Run the Template

Download and extract the Inworld Templates.
Install the Runtime SDK inside the cli directory.
yarn add @inworld/runtime
Set up your Base64 Runtime API key by copying the .env-sample file into a .env file in the cli folder and adding your API key.
.env
```
# Inworld Runtime Base64 API key
INWORLD_API_KEY=<your_api_key_here>
```

Run a basic example of calling the LLM with a text prompt:

yarn node-llm-chat "Hello, how are you?" \
  --modelName=gemini-2.5-flash --provider=google

Now try changing the model and requiring JSON outputs. See Models > Chat Completion for models supported.

yarn node-llm-chat "What is the weather in Vancouver? Return in JSON format" \
  --modelName=gpt-4o --provider=openai --responseFormat=json

Now let’s try with tool calling.

yarn node-llm-chat "What is 15 + 27?" \
  --modelName=gpt-4o --provider=openai --tools --toolChoice=auto

Now let’s try with image inputs:

yarn node-llm-chat "What do you see in this image?" \
  --modelName=gpt-4o --provider=openai \
  --imageUrl="https://upload.wikimedia.org/wikipedia/en/a/a9/Example.jpg"

Finally, check out your captured traces in Portal!

Understanding the Template

The main functionality of the template is contained in the run function, which demonstrates how to use the Inworld Runtime to generate text using the LLM node. Let’s break it down into more detail:

1) Initialize LLM node

First, we initialize the LLM node

import {
  GraphBuilder,
  GraphTypes,
  RemoteLLMChatNode,
} from '@inworld/runtime/graph';

const llmNode = new RemoteLLMChatNode({
  stream,
  provider,
  modelName,
});

When creating the node, you can specify:

provider: The LLM service provider (inworld, openai, etc.) as specified here
modelName: Any model from Chat Completion
stream: Whether to enable streaming responses
textGenerationConfig: LLM generation parameters. You can learn more about these configurations here.

For convenience, createRemoteLLMChatNode does not require registering the component explicitly, before use in the node, but you can also register the component and reference it when creating the node as shown in the node_llm_chat_explicit_components.ts example. This allows you to reuse the same component across multiple nodes (for example, if you want to make multiple LLM calls with the same model).

  const llmComponent = new RemoteLLMComponent({
    provider,
    modelName,
    defaultConfig: TEXT_CONFIG_SDK,
  });

  const llmNode = new RemoteLLMChatNode({
    llmComponent,
    stream,
  });

2) Graph initialization

Next we create a new graph and add the LLM node, setting it as the start and end node. In more complex applications, you could connect multiple LLM nodes to create a processing pipeline.

const graph = new GraphBuilder({
  id: 'node_llm_chat_graph',
  enableRemoteConfig: false,
  apiKey,
})
  .addNode(llmNode)
  .setStartNode(llmNode)
  .setEndNode(llmNode)
  .build();

The GraphBuilder configuration includes:

id: A unique identifier for the graph
enableRemoteConfig: Whether to enable remote configuration (set to false for local execution)
apiKey: Your Inworld API key

3) Graph input preparation

Now we create the message to send to the LLM based on the user’s input.

let graphInput;

if (tools) {
  graphInput = createMessagesWithTools(
    prompt,
    toolChoice,
    imageUrl,
    toolCallHistory,
  );
} else {
  graphInput = createMessages(prompt, imageUrl, toolCallHistory);
}

if (responseFormat) {
  graphInput.responseFormat = responseFormat;
}

Below is an example of what different inputs might look like:

// Basic text message
const basicInput = {
  messages: [
    {
      role: 'system',
      content: 'You are a helpful assistant that can use tools when needed. When analyzing images, describe what you see and use appropriate tools if calculations or weather information is needed.'
    },
    {
      role: 'user', 
      content: prompt
    }
  ]
};

// Multimodal message with image
const multimodalInput = {
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: prompt,
        },
        {
          type: 'image',
          image_url: {
            url: imageUrl,
            detail: 'high',
          },
        },
      ],
    }
  ]
};

// Messages with tool support
const toolInput = {
  messages: [...],
  tools: [
    {
      name: 'calculator',
      description: 'Evaluate a mathematical expression',
      properties: {
        type: 'object',
        properties: {
          expression: {
            type: 'string',
            description: 'The mathematical expression to evaluate',
          },
        },
        required: ['expression'],
      },
    },
    {
      name: 'get_weather',
      description: 'Get the current weather in a location',
      properties: {
        type: 'object',
        properties: {
          location: {
            type: 'string',
            description: 'The city and state, e.g., San Francisco, CA',
          },
        },
        required: ['location'],
      },
    },
  ],
  toolChoice: {
    choice: 'auto' // or 'required', 'none', or specific function name
  }
};

4) Graph execution

Execute the graph with the prepared input:

const { outputStream } = graph.start(new GraphTypes.LLMChatRequest(graphInput));

5) Response handling

Handle responses, including streaming responses.

for await (const result of outputStream) {
  await result.processResponse({
    Content: (response: GraphTypes.Content) => {
      console.log('📥 LLM Chat Response:');
      console.log('  Content:', response.content);
      
      // Handle tool calls if present
      if (response.toolCalls && response.toolCalls.length > 0) {
        console.log('  Tool Calls:');
        response.toolCalls.forEach((toolCall, index) => {
          console.log(`    ${index + 1}. ${toolCall.name}(${toolCall.args})`);
          console.log(`       ID: ${toolCall.id}`);
        });
      }
    },
    ContentStream: async (stream: GraphTypes.ContentStream) => {
      console.log('📡 LLM Chat Response Stream:');
      let streamContent = '';
      const toolCalls: { [id: string]: any } = {};
      let chunkCount = 0;
      
      for await (const chunk of stream) {
        chunkCount++;
        if (chunk.text) {
          streamContent += chunk.text;
          process.stdout.write(chunk.text);
        }
        
        // Accumulate tool calls from stream
        if (chunk.toolCalls && chunk.toolCalls.length > 0) {
          for (const toolCall of chunk.toolCalls) {
            if (toolCalls[toolCall.id]) {
              toolCalls[toolCall.id].args += toolCall.args;
            } else {
              toolCalls[toolCall.id] = { ...toolCall };
            }
          }
        }
      }
      
      console.log(`\nTotal chunks: ${chunkCount}`);
      console.log(`Final content length: ${streamContent.length} characters`);
      
      const finalToolCalls = Object.values(toolCalls);
      if (finalToolCalls.length > 0) {
        console.log('Tool Calls from Stream:');
        finalToolCalls.forEach((toolCall, index) => {
          console.log(`  ${index + 1}. ${toolCall.name}(${toolCall.args})`);
          console.log(`     ID: ${toolCall.id}`);
        });
      }
    },
    default: (data: any) => {
      console.error('Unprocessed response:', data);
    },
  });
}

​Run the Template

​Understanding the Template

​1) Initialize LLM node

​2) Graph initialization

​3) Graph input preparation

​4) Graph execution

​5) Response handling

Run the Template

Understanding the Template

1) Initialize LLM node

2) Graph initialization

3) Graph input preparation

4) Graph execution

5) Response handling