C++ strtok()

#include <cstring>
#include <iostream>
using namespace std;

int main() {
  char quote[] = "Remember me when you look at the moon!";

  // break the string when it encounters empty space
  char* word = strtok(quote, " ");

  cout <<  word;

  return 0;
}

// Output: Remember

strtok() Syntax

The syntax of strtok() is:

strtok(char* str, const char* delim);

In simple terms,

• str - the string from which we want to get the tokens
• delim - the delimiting character i.e. the character that separates the tokens

strtok() Parameters

• str - pointer to the null-terminated byte string (C-string) to tokenize
• delim - pointer to the null-terminated byte string that contains the separators

strtok() Return Value

The strtok() function returns:

• the pointer to the next token if there is any
• NULL value if no more tokens are found

strtok() Prototype

The prototype of the strtok() function as defined in the cstring header file is:

char* strtok(char* str, const char* delim);

Multiple Calls of strtok()

strtok() can be called multiple times to obtain tokens from the same string. There are two cases:

Case 1: str is not NULL

• This is the first call to strtok() for that string.
• The function searches for the first character that is not contained in delim.
• If no such character is found, the string does not contain any token. So a null pointer is returned.
• If such a character is found, then from there on the function searches for a character that is present in delim.
  • If no separator is found, str has only one token.
  • If a separator is found, it is replaced by '\0' and the pointer to the following character is stored in a static location for subsequent invocations.
• Finally, the function returns the pointer to the beginning of the token.

Case 2: str is NULL

• This call is considered as a subsequent call to strtok() with str.
• The function continues from where it left in previous invocation.

Example 1: C++ strtok()

#include <cstring>
#include <iostream>
using namespace std;

int main() {
  char quote[] = "Remember me when you look at the moon!";

  // break the string when it encounters empty space
  // str = quote, delim = " "
  char* word = strtok(quote, " ");

  cout << "token1 = " << word << endl;

  // get the next token i.e. word before second empty space
  // NULL indicates we are using the same pointer we used previously i.e. quote
  word = strtok(NULL, " ");

  cout << "token2 = " << word;

  // get the third token
  word = strtok(NULL, " ");

  cout << "token3 = " << word;

  return 0;
}

Output

token1 = Remember
token2 = me
token3 = when

In this program,

• We have tokenized the quote C-string with an empty space " " as the delimiting character delim. This separates quote into tokens the every time strtok() encounters a space " ".

• The first time we call the function, we need to pass the quote string to specify that it is the source string parameter src.
  
  

  char* word = strtok(quote, " ");

• All subsequent calls to the strtok() function take NULL as the src parameter. This argument instructs the compiler to use the previously-used pointer to C-string (i.e. quote) as the source src.
  
  

  word = strtok(NULL, " ");

Example 2: Print All Tokens in a String

#include <cstring>
#include <iostream>
using namespace std;

int main() {
  char str[] = "Remember me when you look at the moon!";
  char delim[] = " ";

  cout << "The tokens are:" << endl;

  // tokenize str in accordance with delim
  char *token = strtok(str,delim);

  // loop until strtok() returns NULL
  while (token)  {

    // print token
   cout << token << endl;
   
    // take subsequent tokens
    token = strtok(NULL,delim);
  }

  return 0;
}

Output

The tokens are:
Remember
me
when
you
look
at
the
moon!