Navigating new product development: Uncovering factors and overcoming challenges for success

2.1. Top management commitment

Top managers or decision-makers are the ones who give a green or red light when a proposal is presented. This is the very first stage of product development that involves getting approval from the top management. If top management is closely involved in the early stages of NPD, the impetus behind new product ideas is greater. Senior management has the ability to offer resources and clarify project goals. For instance, senior managers must act as process supporters in approving, allocating, and directing the flow of the process [8]. Individual actions that cross functional boundaries can be coordinated by management.

The importance is highlighted in a paper by Ref. [9] where they investigated the Hong Kong toy sector. The critical success factors (CSFs) were examined in the four stages of the NPD process, and significant success factors were divided into four groups according to their implementation and significance. During phase I of the process of developing a new toy, it was observed that top management support and capital backing were two amongst the most frequently employed CSFs [9]. In the product development phase II, senior management commitment is still among the most widely adopted CSFs.

2.2. Involvement of cross-functional team

Front-end success has been characterized as requiring cross-functional collaboration [10]. One possible explanation is that cross-functional collaboration facilitates thorough analysis and reduces front-end volatility. Another possibility is that concept selection is usually conducted in meetings including representatives from various departments in the company [11]. In such gatherings, cross-functional collaboration facilitates concept assessment. Several scholars have looked at different types of cross-functional collaboration. The R&D and marketing interaction, according to Ref. [12], is the most likely instance of cooperative interdependence during the early stages of NPD. These two regions are in charge of product definition and concept, which are then distributed to the rest of the company's activities and departments. Process and manufacturing design should collaborate early in the process of NPD to ensure that the suggested items can be manufactured [13,14].

[15] found that the strength of NDP processes in the food business is largely dependent on customer involvement, exchange of information with internal and external stakeholders, and the importance of having a well-defined NPD process strategy and operational thinking.

2.3. Placement of structured NPD process

The importance of client interaction in the early stages of NPD is debatable. Some observers say, for example, that customers rarely offer corporations substantial or diversified information. However, the production of incorrect product is one of the main causes of NPD failure [8]. Paying close attention to the market's new requirements can give businesses ‘first-movers' advantage, which translates to high product success rates in the face of low competition. Teams that do not integrate client feedback into their product development efforts are likely to fail. Before starting product creation, companies should investigate the expectations and requirements of the client.

Before you take any large steps in business, make sure there is a market for what you are selling. There are no sales if there are no customers. Market research is a method of collecting data and information about your target audience in a methodical manner. It helps in determining the viability of your product or service before launching it on the market. It also offers you a sense of what is hot in the business and what drives consumers to convert and buy. As a result, you can plan your product or service's roadmap.

Venturing into the digital space of online customer reviews [16], harness the power of business intelligence to decipher and decode consumer sentiments. Their findings illuminate the myriad factors that shape consumer perceptions, drawing attention to the significance of online feedback in the product development landscape. In an era where digital voices resonate loudly, their methodology offers a compelling lens to view and evaluate customer preferences.

2.4. Project management capability

The project manager is responsible for the project's progress through its different stages, one of which is the beginning stage, which is divided into several goals to be achieved. A project manager seeks assistance, makes resource requests, and handles technological and organizational challenges. At successful firms, project managers are responsible for all of these responsibilities, according to Ref. [17]. Project managers are also responsible for defining goals, prioritizing work, and providing leadership on the front end. Product definitions are influenced by project managers.

Although no comprehensive study of the characteristics of good front-end project management has yet been conducted, existing research indicates that front-end activities can vary substantially in terms of sequencing, amount of similarity, and duration of the relative period [18,19]. This requires the front-end project manager to possess a wide range of skills.

2.5. Innovative ideas and service innovation

Offering a synthesized overview of new service development literature, Kitsios and Kamariotou, (2020) provide valuable insights into the trajectory and nuances of service-centric innovation. By mapping the extensive landscape, their work acts as a compass, directing firms to the key considerations, challenges, and opportunities inherent in the service development domain.

According to Ref. [20], technological advances enable the development of novel products. However, as a result of this evolution, it is difficult to generate new ideas for the organization. Nonetheless, capitalizing on recent technological advancements is critical to the successful production of a noble product [21]. Not only should ideas disrupt established paradigms, but product representations of these concepts must really bring value to customers to sell [21].

In the realm of service innovation [22], emphasize the critical role of digitization in shaping the innovation process. Their investigation spotlights both areas ripe for exploitation and avenues that beckon deeper exploration. With the digital landscape evolving at an unprecedented rate, the paper underscores that the key to effective service innovation lies in leveraging digital tools and platforms, aligning strategies with emerging technological trends, and recognizing the transformative potential of digitization.

Prior to producing a unique product description, the team gathers market needs from a range of sources [23,24]. Finally, a prototype concept idea is designed, allowing companies to assess whether extra work is required. If the choice is made to proceed, the first product concept enables the development phase's activities to be prioritised [12,17]. A drawing, a diagram, a prototype, or a mock-up can all be used to show an early concept for a product development [25].

3.1. Internal challenges: complexity of teams’ project

Organizations form teams when a single individual or a group of individuals working sequentially cannot adequately complete tasks in a timely manner. To develop new products, teams must navigate an unfamiliar environment fraught with high levels of uncertainty, which serve as crucial drivers of group success [[26], [27], [28]]. On the other hand, when ambiguity and uncertainty are significant, team performance can deteriorate. According to Ref. [27], the contributing factor of ambiguity is twofold: the platform and the market. The first refers to the degree of ambiguity present during the project's design process, whilst the latter refers to the level of uncertainty created by ambiguity over the product's client desires.

According to Ref. [29] advocated regular discussion and courteous engagement, along with other things, to combat individuals' tendency to withdraw from teamwork under stressful conditions. On the other hand, setting acceptable communication styles within authorized teams can be difficult. Numerous interpersonal factors, not the least of which are those related to diversity, lead to the next issue.

3.2. Internal challenges: communicating across functions

While team cross-functionality can produce a range of beneficial outcomes when implemented according to the procedures mentioned in the preceding section, existing research indicates that achieving those outcomes is not simple. Functional variety, according to two distinct assessments of the literature, has a harmful effect on team performance in general, but especially during times of crisis and upheaval [30,31]. Apart from impairing team effectiveness, complexity has been linked to increased levels of discontent, attrition, sick absence, commitment, and workplace stress [32]. Their emphasis on team culture is very appropriate. Prior work by current authors and others established the crucial importance of the personal and social environment [[33], [34], [35]].

Meaningful demographic contrasts elicit judgements about an individual's perceived significance, aptitude, and potential to effectively respond to the job [36]. When these perspectives are wrong or out of control, they obstruct cross-functional learning and collaboration. Due to judgement errors, systems of influence and deference emerge within the team, limiting the supply of job knowledge [37]. Less honoured team members, who are more likely to experience decreased psychological safety [35], decreased self-efficacy, and a diminished sense of importance to the team and its task become less engaged in team tasks [35,38], and thus engage in less team-learning behavior [33,34,39].

3.3. Internal challenges: temporary team membership

Teams are project-based in a wide variety of fields, including research and new product development. When a project or field of inquiry presents itself, organization members are chosen based on their unique capacity to assist the endeavor. Certain members of the team will work on a project until it is completed, and then move on to the next one that requires their specialised skillset; others will work on the project for a shorter period of time. Individuals may collaborate on several projects with different people, depending on the organizational objectives. This flexible structure enables projects to be staffed by the most qualified professionals. At the same time, the transient character of the team might be troublesome; members must get to know one another before they can operate well as a team [40].

Consistent team participation, according to research, increases instructional and intrateam interaction [41]. Individuals who function in teams for an extended period of time, up to three years [42], become more efficient, most likely because teammates develop “transactive memory” [43]. Longevity is crucial for cross-functional collaboration in particular [44]. discovered that social tenure had a moderating effect on the link between diversity and conflict, with conflict being smaller in varied teams with a longer team duration.

[32] discovered that group longevity modifies the connection of diversity to organizational performance in a more recent study. As a result, firms tend to be forced to make a trade-off when utilizing short, project-based teams. On the other hand, they enable the application of the highest level of knowledge to every project. On the other hand, the frequency and duration of these proposal teams preclude the development of familiarity and understanding that comes with team longevity.

3.4. Internal challenges: fluid team boundaries

One reason for categorising a group of individuals as a “functional team” is to ensure that the group is “anchored,” which means that each member's role is explicitly expressed and acknowledged [45]. Individuals who work in constrained teams are more likely to have comparable time allocations than members of ad hoc teams. This enclosure generates a sense of shared identity, cohesiveness, and purpose, all of which contribute to the urge for cooperative behavior [46,[46], [47], [48]].

Two issues arise with NPD teams. To begin with, the advantage of cohesion has restrictions. Members can become so self-absorbed that they lose sight of the outer world and their own connections, compromising team effectiveness. Both environmental [[49], [50], [51]] and border [[52], [53], [54], [55]] perspectives assert that external contact with individuals outside the team makes a significant contribution to team performance [55]. established that, while correlation was used to determine that communication has no effect on team performance, interteam communication does.

Second, NPD teams rarely operate within defined exact parameters and adhere to consistent time restrictions. Typically, NPDs are made up of a core set of members who are fully responsible for group performance and depend on others to fill temporary team positions. As [33] illustrates, an NPD team may include core members from advertising, product engineering, and manufacturing, as well as part-time members with finance or legal expertise. Emphasizing on full squad participation for activities that are organically smaller than core duties is inefficient, which is why only a few NPD teams embody the “real team” notion, hence raising the coordination barrier in favour of more efficient resource utilisation.

3.5. Internal challenges: organizational structure

Numerous organizations fail to develop structures that support the success of teams [56]. Certain organizational structures, such as individual-based awards and department-based sharing of benefits, work against teamwork [57]. Numerous team failures are attributed to inconsistencies in task–reward structures [58]. To maximise performance and productivity, the interrelation of incentives should match the complexity of tasks [59]. Individuals or departments' contributions to the final product are barely distinguishable in the ideal form of teamwork, and as a result, all individuals should get equal credit for their team's play. This does not stop the team's “star” members from receiving further individual recognition [15]. To foster cooperation, however, organizational rewards must be pervasive, acknowledged, and appreciated such that the message communicated to team members is coherent [60].

Regardless of the fact that service innovation has a favorable influence on team performance, many firms set performance evaluation and rewards on an individual basis [59]. This may encourage team members to prioritise individual achievement and credit over collective goals, particularly when they conflict [61]. described a new product development team in which an engineer was responsible for utilizing cutting-edge technological innovation and the marketer was responsible for establishing connections with clients to identify their desires for the new product. Although these foci were consistent with the values and incentives of their functional divisions, they impeded team achievement. Therefore, the challenge to cooperation is that it operates within an environment that promotes personal achievement, but strives for collective success.

3.6. External challenges: price-income levels

The size of their markets justifies international firms' spending in research and development, which results in numerous new product/service developments. When these firms attempt to introduce these new products/services in poor nations, they frequently retain the majority of the product/service attributes, resulting in costs that are relatively costly for the majority of developing country consumers. In the majority of developing countries, only roughly 5–10 % of households are categorised as middle class [13]. Except for China and India, middle-class markets are typically modest. A high price may imply that the product is too advanced for the customer's needs. As a result, it may drive potential consumers to seek more relevant alternatives.

3.7. External challenges: technological-developmental issues

Generally, developing countries lack a strong technology foundation and trained scientists. Due to a lack of funding, many countries have very few research universities, which contributes to the shortage of highly trained scientific experts [13]. Technology is the catalyst for innovation. When a technical base is lacking, including trained researchers and funding, new product development is difficult. It is difficult for multinational corporations to justify investing in human and financial capital just for the purpose of developing products for underdeveloped countries. As a result, these corporations tend to concentrate their efforts on developing products/services for developed countries on markets capable of supporting these breakthroughs and frequently overlook developing markets.

Often, technological flaws in the product are the fundamental reason for its failure. Designers and product technologists are capable of overengineering even the most sophisticated laboratory instruments. This benefits the company's technical advantage over competitors. However, an “over-engineered” product is costly for the company and ultimately for the customer, since competitors gain an edge over the “over-engineered” product. Although technical deficiencies must be remedied, the cost of doing so should not be unacceptably high.

3.8. External challenges: capital constraints

Financial resources are often limited in developing countries, with the available capital primarily directed towards economic development rather than investment in products and services. International corporations primarily generate their revenues from large industrialized markets, making it challenging to justify allocating funds for research in capital-scarce developing nations. Executives of international companies in these countries often face budget constraints for research and development due to the high costs and scarcity of capital. Consequently, these leaders tend to focus their efforts on cost reduction in areas such as manufacturing, shipping, marketing, and customer service [].

Innovative organizations encounter financial barriers when it comes to investing in innovation due to externalities, informational asymmetries, and challenges related to appropriability of returns on R&D investments. These factors contribute to higher costs associated with R&D investments, leading to underinvestment in innovation activities. The cost disparity between external and internal costs can further exacerbate this underinvestment, along with limitations in liquidity. Consequently, some innovative ventures may be halted, delayed, or abandoned due to insufficient financial resources.

4.1. Research design

In addition to conducting literature research, this study incorporates a one-to-one interview with the CEO of a Malaysia-based company who plays a pivotal role in the success of the firm's product and service development. The one-to-one interview method is widely utilized for data collection purposes. Individual interviews are recognized as a valuable approach for gathering comprehensive and in-depth data, providing valuable insights into individuals' perceptions, understandings, and experiences related to a specific phenomenon [62].

4.2. Data collection

Data was collected through an interview with Protenga, a leading insect-based company originating in Singapore and now operating from a pilot facility in Malaysia. Since its inception in 2018, Protenga has garnered international acclaim, carving a niche for itself as a frontrunner in Malaysia's insect industry.

The interviewee, Leo Wein, serves as both the CEO and founder of Protenga. With substantial experience in introducing new products and services to mature markets, Mr. Wein's insights are crucial to this research. As the primary informant for the company, he was the solitary participant in the interview.

Given the safety precautions necessitated by the Covid-19 pandemic, the interview was conducted virtually. A comprehensive questionnaire containing 24 items was emailed to Mr. Wein. The objective of this set of questions was to comprehend the strategies Protenga employed to surmount the hurdles encountered during its nascent stage, and how it successfully launched innovative products into the marketplace.

4.3. Data analysis

The analytical framework for this study will rely on thematic content analysis to dissect the data obtained from the interview. Thematic analysis is a method used in qualitative research to analyze data and identify patterns or themes within it. It is considered a foundational component of qualitative research [63].

Beyond the thematic analysis of primary interview data, the secondary data is organized in a tabulated form with checkboxes in each column. Each area is labeled according to the source of the research paper, and the record consists of a list of aspects mentioned in the publications. By utilizing checkboxes, it becomes possible to determine the variables that were most frequently mentioned as important factors in the success of new product development.

4.4. Background of protenga

Protenga, established in 2016 and headquartered in Singapore, operates from its pilot facility in Johor, Malaysia. The company specializes in the breeding, farming, and production of products derived from the Black Soldier Fly (BSF). In a discussion with Leo, the founder and CEO of Protenga, he delineated the company's aspiration to address the global environmental crisis. Portraying Protenga as a firm that merges insect technology with nutrition, Leo emphasized the deployment of technologically advanced insect farming systems to promote circular nutrient systems. This allows Protenga to manufacture eco-friendly protein for animal feed, as well as pet food.

The company's innovation stems from its use of the BSF lifecycle, establishing a symbiotic relationship between these insects and humans. As Leo elucidated during the interview, these insects have a dual role in the ecosystem - decomposing organic matter at the end of its life cycle and serving as a food source for other creatures. Inspired by this inherent circularity, Protenga developed technology to bring this natural cycle into the human food system. Their aim is to harness this technology to produce sustainable and high-quality feed and food products.

4.5. Protenga businesses

Protenga has diversified its portfolio to include three key products: fertilizer, protein, and oil. These are derived from various life cycles and processes at the company's Black Soldier Fly (BSF) farm, yielding versatile products applicable to industries such as agriculture, livestock, and aquaculture. Protenga's pricing strategy is tailored to each customer order. Products are benchmarked against market equivalents, such as fish meal (RM 4-6 k), fish oil (RM 4-6 k), and composted poultry manure (RM 400–800).

Operating as an integrated entity, Protenga covers the entire value chain, from primary resource production to supplying finished products. Their latest venture, YumGrubs, operates on a Business-to-Consumer/Direct-to-Consumer (B2C/D2C) model, directly selling their fast-moving consumer goods (FMCG) to end users. To reach their target market, digital marketing, e-commerce platforms, and content/social marketing strategies are leveraged. In terms of their Business-to-Business (B2B) offerings, Protenga adheres to traditional marketing strategies, with an emphasis on nurturing personal customer relationships which fosters trust and facilitates the formation of enduring business partnerships.

5.1. Innovative idea

In response to the question, ‘What opportunity did you identify that motivated the inception of an insect-based business?’, the CEO articulated: ‘I envisioned an opportunity to design a technology that would render this natural insect-driven process amenable to the human food system, thus facilitating the production of sustainable, high-quality food and food products.’ Innovations in technology usher in new product development opportunities. The implementation of state-of-the-art technology is integral to the successful creation of superior products. The CEO concurs, asserting that the firm is offering solutions to global issues such as food shortage and excessive waste: ‘Indeed, our approach contributes to circular economy solutions within the food system.'

5.2. Placement of a structured NPD process

The responsibility of management in relation to the venture team is to architect the NPD framework, guidelines, and benchmarks. This strategic planning facilitates clarity for team members regarding their roles and how to navigate the NPD process. Since 2016, Protenga has executed meticulous and organized research and development in fields encompassing biology, entomology, engineering, technology, and of course, business development. The CEO affirmed: “Our production system is grounded in extensive basic and applied research conducted since 2016.” NPD approaches should underscore quality throughout the rollout phase. Moreover, these procedures must display flexibility, with the capability to merge phases, execute them simultaneously, or eliminate them upon thoughtful consideration.

5.3. Involvement of cross-functional teams

In response to inquiries about his management style, the CEO revealed that they have established leadership principles at Protenga.

• ●

  Utilizing Insects to Our Advantage

• ●

  Initiating Actions and Valuing Independence

• ●

  Balancing Freedom and Responsibility

• ●

  Prioritizing Purpose, Humbling Ego

From these principles, they have fostered an encouraging, inclusive, and results-oriented team culture. The CEO further elaborated, ‘We are accountable to each other, maintaining transparency and honesty in our communication.’ Cross-functional venture teams often operate with a degree of autonomy, displaying a mix of entrepreneurial attributes that enhance each other, thereby boosting process performance and outcomes. The variety of perspectives inherent in cross-functional collaboration promotes innovation.

5.4. Challenges: internal challenges

In response to the query regarding current challenges that require future amelioration, the CEO stated, “We have a competitive stance overall. However, being based in Malaysia, we encounter market access hurdles, for instance, to the EU, which we need to tackle in cooperation with the relevant authorities. We are still evolving and enhancing; hence our production volumes require a significant increase to reach their full potential."

For a nascent company like Protenga, constrained resources undoubtedly pose one of the greatest hurdles to scaling up production. This is not only from a financial perspective but also considering intangible assets of the company, such as employee skill sets and human resources.

5.5. External challenges

In discussing the challenges encountered during the export process, the CEO highlighted the complexity of differing rules and regulations imposed by various countries. What is permissible in one country may not necessarily be acceptable in another. It is therefore vital to understand the laws stipulated by the importing country to facilitate a seamless transaction.

In terms of licenses obtained by Protenga, given its production of consumable goods, the CEO confirmed, “We hold a Malaysian manufacturing license, production license for each product, sales license, and local business license. Additionally, we've acquired several free sale certificates for export."

In regards to the attitudes of Malaysians and the government towards food scarcity and excessive waste, and the actions the CEO might suggest, he said, “I'm not in a position to judge the level of seriousness attributed to these issues. However, I do see certain initiatives and policies being developed or already in place, which is promising. For instance, the government agency BioEconomy Corp is fostering the agtech and biotech field in Malaysia. Yet, we certainly hope to see more meaningful and impactful adoption of these initiatives. For instance, mandatory food waste segregation could be a significant step towards a sustainable food system, but we are still a considerable distance from that in Malaysia."

Food scarcity and excessive waste are globally critical issues, and Malaysia should give them due attention before its too late. In Europe, government approval has been granted for the use of insect proteins in pig and poultry production [64]. However, no such rules exist in Malaysia permitting the use of processed protein in any industry [65]. highlighted the challenges of producing insects for human consumption, emphasizing that stringent food safety and standard protocols must be followed. The establishment of a food-grade production facility for insect protein can be costly, and compliance certification presents a significant hurdle. Given the substantial investment and lower demand, many insect growers continue to produce animal feed and fertilizer. Therefore, if one were to produce insects fit for human consumption, food safety control is crucial, requiring the prudent handling of non-contaminated food surplus and the elimination of food waste as feed.

Table 1 represents the distribution of these critical success factors in the chosen research papers. These factors are arranged in descending order, from the most frequently discussed to the least. Top management commitment during the new product development (NPD) process emerged as the most frequently discussed factor. It is highlighted in all five selected papers, emphasizing the crucial role that top management's involvement plays in providing professional insights during the NPD process. Four factors - specific goals and milestones, cross-functional team participation, talented team members with relevant NPD experience, and a clearly defined product concept - tie for the second most frequently discussed topics, appearing in four out of the five papers. Three papers highlighted the importance of fostering an entrepreneurial culture and including user and customer participation in the NPD process. Alignment of NPD activities with overarching strategy and the availability of financial reports were each cited twice across the papers. Finally, factors such as a structured NPD process, efficient intra-organizational communication, a focus on innovation and unique ideas, and the pace of the NPD process were each mentioned once in the studied papers. Factors and challenges for product development are illustrated in Fig. 1 below.

6.1. Dual-domain model of NPD challenges

One of the distinctive contributions of this paper is the identification and classification of challenges into two distinct domains: Internal and External. The Dual-Domain Model underscores the multifaceted nature of NPD, suggesting that companies must be adept at navigating challenges both from within their organizational structures and from the broader external business environment.

6.2. The NPD continuum of collaboration

This paper highlights the varying degrees of collaboration within NPD teams, from cross-functional collaboration to the dynamic nature of team boundaries. This continuum showcases that collaboration isn't binary; it's a spectrum. At one end, we have fixed, cross-functional teams, and on the other, fluid teams with changing membership. Recognizing where a team stands on this continuum can offer insights into potential challenges and the strategies needed to address them.

6.3. The sector-specific lens

NPD doesn't operate in a silo; its success and challenges are inherently tied to the sector it belongs to. By emphasizing the need to understand sector-specific variables, this paper proposes that NPD strategies must be tailored, adaptive, and sector-aware. This brings forth the idea that generalized NPD strategies might be less effective than previously thought.

6.4. The NPD success factors

This paper not only identifies challenges but also alludes to critical success factors. By juxtaposing challenges against success factors, we propose the NPD success model. This model can serve as a diagnostic tool for firms, helping them identify areas of strength and potential pitfalls based on internal dynamics and external market conditions.

6.5. The external challenge

Our findings unearth external challenges. While external factors such as technological-developmental issues can pose challenges, they can also serve as opportunities for firms to innovate and lead. Recognizing and acting upon these paradoxical opportunities can be a game-changer for firms in the NPD realm.

6.6. The organizational inertia dilemma

While organizational structures provide stability, they can also hinder agility, especially in the dynamic world of NPD. This paper surfaces the tension between stability and agility, suggesting that companies need to strike a balance. Too much rigidity can stifle innovation, while excessive flexibility can lead to a lack of direction.

By conceptualizing the findings in the manner above, this paper not only provides an in-depth understanding of NPD but also advances the discipline by offering novel frameworks and perspectives (see Fig. 1). These insights, grounded in rigorous qualitative analysis, have the potential to reshape how academics and practitioners approach NPD in the future.